Consuming Topic in Apache Beam - elements don't get processed

Hi there,
I’m trying to integrate my Apache Beam pipeline with Confluent Kafka yet I’m unable to have messages processed.

My pipeline is using Beam’s Python SDK and I’m using the package.

I built a simple pipeline to test the integration and I’m using the DatagenSourceConnector to generate random test data.

My code:

with beam.Pipeline(options=self.beam_options) as pipeline:
            kafka_elements = (
                | "Consume from Kafka"
                >> ReadFromKafka(
                        "bootstrap.servers": "SERVER:9092",
                        "security.protocol": "SASL_SSL",
                        "sasl.jaas.config": " required username='USER_NAME' password='PASSWORD",
                        "sasl.mechanism": "PLAIN",
                        # Required for correctness in Apache Kafka clients prior to 2.6
                        "client.dns.lookup": "use_all_dns_ips",
                        # Best practice for higher availability in Apache Kafka clients prior to 3.0
                        "": "45000",
                | "Print elements" >> beam.Map(print)

When I run the pipeline I can see that the consumers are updating their offsets yet nothing is printed. I even tried using a different method instead of ‘print’ where I just throw an exception but that code never triggers.

Any insight on what I’m missing here would be highly appreciated :pray: