KSQL Migrations Tool

[kjang1750]

Hey all,

I have been playing around the ksql-migrations tool. So far things are going pretty well with queries being deployed while the feature for “CREATE OR REPLACE STREAM” has helped me with the Source Control Process.

Currently, I am able to have multiple teams on it, but when it comes to automation it seems tricky due to the fact that the files need to be in a specific format seen below

V_000000X_streams.sql
The steps I have done to migrate the streams are below
##HMM, so in the folder “ksql-migrations-gitops”, I created a folder for our group called team_a and team_b. We can create streams following these protocols

sudo mkdir team_a

sudo chmod -R 777 team_a

1. ksql-migrations new-project ./ksql-migrations-gitops/team_a http://localhost:8088

—modify properties file----

2. ksql-migrations -c ./ksql-migrations-gitops/team_a/ksql-migrations.properties initialize-metadata

3. ksql-migrations -c ./ksql-migrations-gitops/team_a/ksql-migrations.properties create streams

4. ksql-migrations apply -a -c ./bhg-ksql-kaas/team_a/ksql-migrations.properties

----------------------END-----------------------------------------

----NEW TEAM ONBOARDING--------------------

#Let me try to create another one

sudo mkdir team_b

sudo chmod -R 777 team_b

1. ksql-migrations new-project ./ksql-migrations-gitops/team_b http://localhost:8088

----modify the properties file see “ksql-migrations.properties”----

2. ksql-migrations -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties initialize-metadata

(Why do my migration tables and streams need to be different? How can this be automated if I need to create the file in the specific format first?)

3. ksql-migrations -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties create streams

4. ksql-migrations apply -a -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties
   

   

It seems like any sort of automation and source control in ensuring KSQL queries can be created may be tough despite the migration tool being pretty useful. I have my directory structure shown below. Luckily if this was manual, it would be easy. Has anyone had any automation methods when it comes to using the KSQL Migration tool? Thanks!

Update: for the .sql file, looks like the name of the file can be whatever we want as long as we follow the format “V00000X__<any_name>.sql” where X = the version number. It is still a bit odd that I am using multiple Migration KSQL Streams and a Migration Schema Versions though. Has anyone had experience with having multiple teams have KSQL in their source control with only one Migration KSQL Stream and Schema Versions for each environment?