I have been playing around the ksql-migrations tool. So far things are going pretty well with queries being deployed while the feature for “CREATE OR REPLACE STREAM” has helped me with the Source Control Process.
Currently, I am able to have multiple teams on it, but when it comes to automation it seems tricky due to the fact that the files need to be in a specific format seen below
The steps I have done to migrate the streams are below
##HMM, so in the folder “ksql-migrations-gitops”, I created a folder for our group called team_a and team_b. We can create streams following these protocols
sudo mkdir team_a
sudo chmod -R 777 team_a
- ksql-migrations new-project ./ksql-migrations-gitops/team_a http://localhost:8088
—modify properties file----
ksql-migrations -c ./ksql-migrations-gitops/team_a/ksql-migrations.properties initialize-metadata
ksql-migrations -c ./ksql-migrations-gitops/team_a/ksql-migrations.properties create streams
ksql-migrations apply -a -c ./bhg-ksql-kaas/team_a/ksql-migrations.properties
----NEW TEAM ONBOARDING--------------------
#Let me try to create another one
sudo mkdir team_b
sudo chmod -R 777 team_b
- ksql-migrations new-project ./ksql-migrations-gitops/team_b http://localhost:8088
----modify the properties file see “ksql-migrations.properties”----
- ksql-migrations -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties initialize-metadata
(Why do my migration tables and streams need to be different? How can this be automated if I need to create the file in the specific format first?)
ksql-migrations -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties create streams
ksql-migrations apply -a -c ./ksql-migrations-gitops/team_b/ksql-migrations.properties
It seems like any sort of automation and source control in ensuring KSQL queries can be created may be tough despite the migration tool being pretty useful. I have my directory structure shown below. Luckily if this was manual, it would be easy. Has anyone had any automation methods when it comes to using the KSQL Migration tool? Thanks!
Update: for the .sql file, looks like the name of the file can be whatever we want as long as we follow the format “V00000X__<any_name>.sql” where X = the version number. It is still a bit odd that I am using multiple Migration KSQL Streams and a Migration Schema Versions though. Has anyone had experience with having multiple teams have KSQL in their source control with only one Migration KSQL Stream and Schema Versions for each environment?