Hi all! I have a pretty niche setup I need to get going and I have not been able to find any specific documentation/threads that cover it. Hopefully someone out there can help
To keep it brief, I am attempting to write a Docker Compose that can spin up 1 Zookeeper with 2-3 Brokers that is using SASL_PLAINTEXT with SCRAM-SHA-256 as the consumer authentication. This should also be done using a JAAS config file for specifying the username and password. The version must also be 5.5.X for the Confluent images.
Why this specific setup you might ask? I am developing a service to connect to an environment that already exists in a customer’s premise. This is to get a replicated setup that my team can utilise for development and prototyping, as it is our first time getting hands on with Kafka.
I think I have turned every link in Google purple at this stage, and read over pages and pages of Confluent documentation, but nothing out there seems to combine all of these things in one go. The main things tripping me up are which environment variables are actually exposed for this version and how to format them (in regards to the JAAS config, specifying SCRAM and setting up the protocols for the consumers etc.).
This is kind of where I’m at with it now, as you can tell I’ve started to Frankenstein in the SASL_PLAINTEXT and JAAS, not sure if this is on the right track (please correct my mistakes). I’m confused by how to mention the SCRAM mode, because I’ve seen it implemented by overriding the startup command and sometimes by using env variables… it’s a bit inconsistent.
version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:5.5.6 hostname: zookeeper container_name: zookeeper restart: unless-stopped ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker-1: image: confluentinc/cp-kafka:5.5.6 hostname: broker-1 container_name: broker-1 depends_on: - zookeeper restart: unless-stopped ports: - "9092:9092" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker-1:29092,EXTERNAL://localhost:9092 KAFKA_DEFAULT_REPLICATION_FACTOR: 2 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_NUM_PARTITIONS: 3 KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/kafka_server_jaas.conf" volumes: - "./secrets:/etc/kafka/secrets"
If you know how to set up something like this, please reach out to me I haven’t joined the Slack yet but if you want to share your knowledge over a more reactive messaging platform (Discord also works), let me know. Otherwise sharing this knowledge with other community members will help also!
I’ll try report back here if I figure it out myself!