Sending data to Kafka from devices (e.g. Raspberry Pi, Tinkerboard)

[accat]

Hello everyone,
I followed this beautiful guide to install kafka on GCP:

Setting up your first Apache Kafka development environment in Google Cloud in...

I have followed all the steps and everything works fine. I was wondering: I have a lot of devices around the world that would like to send data to Kafka on my GCP. How can I do it? does it make sense to use Kafka in this way?

Thanks

[gianlucanatali]

Hi @accat , welcome to the forum! This is a very interesting topic, general answer is: yes, it is possible and many companies do that. The how, really depends on your use case. I can point you to some blog posts that will be able to give you an overview way better than I could in this message Feel free to come back here and add more details for your use-case and let us know if this info was useful!

• IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video
• Use Cases and Architectures for Kafka at the Edge. This could be interesting if you also want to run kafka at the edge .

[accat]

They are truly “theoretical” posts, I was looking for something more practical with examples of sending data from devices in the field to Kafka

[rmoff]

What kind of devices? Practically you’d probably end up using MQTT or maybe HTTP, both of which are supported in the Kafka ecosystem. If you can give us more specifics to go on we can help you better

[accat]

@rmoff I use Raspberry and TinkerBoard with NodeJS or Python environment.
Yes, with MQTT or HTTP protocol. How can I send data from these devices to Kafka deployed on my GCP?

[gianlucanatali]

Hi @accat , then maybe this example could be interesting?

• If you want to try Confluent Cloud, there is a full set of clients examples ready to use, Including a Python example

Some more sophisticated demos using specific MQTT technologies

• Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorFLow

• You can use Confluent MQTT Proxy, see tutorial

• this can be another interesting example, using waterstream.io: Waterstream Fleet Demo. Blog post if you are interested in higher level architecture

[accat]

With MQTT proxy can I send data from a remote device (for example a Raspberry somewhere in the world) to a Kafka cluster installed on my GCP?
Thanks

[gianlucanatali]

Sure. Probably you will install the proxy somewhere reachable from you devices, and the proxy should have access to Kafka too. Proxy will receive MQTT from your devices and produce to Kafka using the producer API.

[accat]

I’ll try and I’ll give you a feedback!

[accat]

I finally found the time to try something.
But I don’t understand one thing.
Reading this guide I should have an MQTT Proxy that I don’t actually have inside Kafka …
How do i install it?
Thanks