How to get data from snowflake in kafka topic in confluent UI? Any idea?
Hi,
afaik there is no Snowflake source connector at the moment.
The JDBC connector might be a solution to try.
out of curiosity:
why do you need to load the data from Snowflake to Kafka?
is it possible to load the data loaded to Snowflake also to Kafka ?
Best,
Michael
I’d need more information to get more specific but in a general sense, with SLAs measured in minutes/hours.
what I’ve seen done is use Snowflake’s COPY TO command Overview of Data Unloading — Snowflake Documentation to push the data to S3/google blob, then use the S3 source connector or just a process coordinator like airflow, to scrape S3 and push it to kafka. This isn’t a great solution and it’s not really streaming, it can also be costly on the snowflake resources.
Your better option is to fork the pipeline before it hits snowflake and create a standard ‘lambda architecture’. Which I think is what @mmuehlbeyer is saying.
1 Like