Priority and partitions for messages

Kafka Connect
1

Hi,
We have different types of messages. Some are security alerts to be pushed to devices and others are alerts for billing cycles, Standing orders and announcements. Moreover all messages are not uniformly handled by the consumer. Some are push notifications and others are routed to the Elastic search Kafka Connector Sink.

Kafka JDBC connectors are what we are looking at to pick up DB changes.

So I am looking for some material to read to understand how to partition and cluster. I will be able map the general guidance to our requirements.

  1. How do I handle security alerts as high priority messages ?
  2. Can Kafka JDBC connectors be scheduled to pick up from the DB ? I could have a clustered scheduler to send to the producer as an alternative.

Thanks

2

Hi there! I’d need a little more information to answer 1. Are all of these messages being picked up from the database or are they from different sources?

Re: 2. JDBC source connectors can be scheduled in that you can tell them to run every 5 mins, 30 sec, etc.

3

Kafka doesn’t have “priorities” like other messaging systems do. What we do for kafka is create different topics for different message types and route the messages appropriately. Also because kafka doesn’t really have a limitation on the number of consumers we simply designate a consumer group to read and act on those high priority messages.

The JDBC connector has poll.interval.ms which allows you to tune how frequently the JDBC source connector polls for new rows. But it does not contain a “scheduler”.
You can schedule fetches with pause/resume and bulk mode but that is probably not the path you want to go down.

1 Like
4

Hello,
The source is primarily the DB and the scheduler that has to fire on bill due dates to pick data from the DB. But the sinks are different.

  1. Elastic Sink
  2. Kafka consumer logic to push to Firebase to device.
  3. Kafka consumer logic to send SMS

This is where the consumer groups could help.
And the data is based on the domain as I mentioned earlier.

  1. Security alerts
  2. EMI alerts
  3. Standing order alerts
  4. Announcements
    etc.

Thanks.

5

I did come across consumer groups. Should I then look at domain-oriented topics and consumer groups based on the target and types of messages ?

Thanks.

6

You should for sure look into consumer groups and topics. Those two things are fundamental to architecting and developing with Kafka.

"The Magical Rebalance Protocol of Apache Kafka" by Gwen Shapira - YouTube is a great into to consumer groups.

7

That was a good overview. Consumer groups for Kafka Connect and other groups based on what the consumers do can be set up. A SMS system interaction in the consumer is a separate group from a Push notification group. Is a group based on what happens in the consumer ?
Can we have DLQs for each consumer group based on whether an SMS is sent or a Firebase Push is done ?

8

@mohanr Yes you can but you’ll have to do the codeing for the DLQ your self. Kafka doesn’t have a traditional DLQ functionality like other messaging systems.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.