Datagen container vs datagen source connector

Ops Containers
1

Hi All

See the newest docker-compose.yml comes with a datagen container, is there a “how to use/manage/configure” this.
that defines what the different types of data it can generate, and where to send it.
vs
@rmoff examples where he goes into source connector configuration to configure datagen.

– assume if you want multiple feeds running at the same time you can start this container multiple times, each with it’s own configuration –

G

2

Could you please post the container name or link the docker-compose you’re referring too? i’m going to assume it’s GitHub - confluentinc/kafka-connect-datagen: Connector that generates data for demos. In that case you find config examples at kafka-connect-datagen/config at master · confluentinc/kafka-connect-datagen · GitHub and in the README.md

3

will go look at those links, thanks.

newest cp-all-in-one docker-compose.yml

  ksql-datagen:
    image: confluentinc/ksqldb-examples:6.2.0
    hostname: ksql-datagen
    container_name: ksql-datagen
    depends_on:
      - ksqldb-server
      - broker
      - schema-registry
      - connect
    command: "bash -c 'echo Waiting for Kafka to be ready... && \
                       cub kafka-ready -b broker:29092 1 40 && \
                       echo Waiting for Confluent Schema Registry to be ready... && \
                       cub sr-ready schema-registry 8081 40 && \
                       echo Waiting a few seconds for topic creation to finish... && \
                       sleep 11 && \
                       tail -f /dev/null'"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      STREAMS_BOOTSTRAP_SERVERS: broker:29092
      STREAMS_SCHEMA_REGISTRY_HOST: schema-registry
      STREAMS_SCHEMA_REGISTRY_PORT: 8081
4

got the datagen container working, pushing page views to the topic.
thanks.

started data generation using:

curl -X POST -H "Content-Type: application/json" --data @config/connector_pageviews.config http://localhost:8083/connectors

and view activity

docker-compose exec connect kafka-console-consumer --topic pageviews --bootstrap-server kafka:29092  --property print.key=true --max-messages 5 --from-beginning

how do I stop the connector/data generator from generating data though ?
I’ve tried just stopping the sql-datagen container and that did not kill the generation.

G

5

answering own question, I can pause it in the control centre and I can delete it in the control centre also.

G