Hi,
New to Kafka and new to the community.
We are currently in the development phase of using Kafka and have been learning a lot about the technology. We have a multi-tenancy setup due to the nature of the data we handle and this results in a number of Oracle databases. This is circa 100 Oracle databases.
We have done some development work with consumers/producers using JDBC connectors to prove the process works. We are now agreeing the design for the multiple Oracle connections we need and have a number of approaches we can take but want to get some feedback on the community first. We have done similar projects in the past with RabbitMQ and have been burnt by the management of a large number of consumers. So at a very high level I believe we have two main design options:
- A consumer per database which means a 1:1 mapping with a topic and thus topics to the same scale. The main disadvantage I’m seeing here is having at least 100 consumers to manage. The main advantage being the connection to the db is kept open.
- A consumer that is able to create and close dynamic connections to the 100 databases we need to connect to and insert data. Based on data in the topic it could create the connections dynamically. Main disadvantage here is having to constantly open/close connection per event as well as needing to write a custom connector. Main advantage is having a small number of consumers to manage.
I appreciate its a very high level overview of the situation but it will have a large impact on our implementation if we get the design wrong. Thanks for any support or similar experiences shared.
Nick