I have my IoT devices to send messages to a MQTT broker cluster. I am planning to streamline packets from mqtt brokers to kafka brokers and thinking of utilizing Kafka Connect MQTT source connector. Yet, almost all references I found on the usage of mqtt connector use single connector with max.Tasks=1, meaning that there is only one task subscribing to the topic. Yet, when message throughput is high on a single topic, I think scaling out connector tasks is a nice approach, but simply spawning more tasks will result in redundant messages since tasks subscribing to the same topic will all get the messages from the broker cluster.
Now I am curious what the best practice for ingesting mqtt messages to kafka brokers in a scalable fashion.
Can you take advantage of hierarchy to have multiple mqtt.topics values? The connector supports single and multiple level wildcards (+ and # respectively). If you’re able to use wildcards to cover the space of the one topic with multiple mqtt.topics values, then you can scale up to as many active tasks as there are comma-separated values.
This isn’t correct – the connector assigns tasks by mapping the comma-separated values in mqtt.topics to tasks, and there’s no overlap of assignments. Each value only goes to one task as described here. As those docs say, though, the connector may output duplicates since it offers at least once delivery; just keep in mind that those dupes aren’t because the same topic gets subscribed by multiple tasks.
Thanks for the information. This is very useful.
So can I just assume that one topic (either wildcard or single) is assigned per task by default? What would be the best approach to increase throughput on one topic then?