we are using confluent kafka platform and java to stream binary messages from IOT devices to the kafka.
The IOT devices encrypts the part of a message using an aes key. The java app needs to do custom security handshake and get the key as a message from the kafka topic and do decryption, if the security handshake exchange fails ,then IOT will never place the data until the handshake is successful.
flow:-
iot (AMQP queue starts handshake process when client connects or amqp broker reconnects.) —> gateway (dumb one K8S java app write to kafka topic) —> decoder( K8S kstream app decodes as per avro schema,decrypted sub message) —> decryption(? where to fit in /handshake message can come at anytime in this topic/ combine logic in decoder ) ----> sub msg decoder(parse business events) —> downstream app
diagram:-
Design:-
my initial design is to use Request async to do in a separate flow and then re flow the unencrypted message to sub msg decoder. how can better be implemented in Kafka