Error when trying to create a connector with curl request

Hello everyone,

I’m following the tutorial by Robin Moffat Loading CSV data into Kafka running docker in a windows machine.

When i launch this curl request:

curl -i -X PUT http://localhost:8083/connectors/source-csv-spooldir-00/config -H ‘Accept:application/json’ -H ‘Content-Type:application/application/json’ -d ‘{“connector.class”: “com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector”,“topic”: “orders_spooldir_00”,“input.path”: “/data/unprocessed”,“finished.path”: “/data/processed”,“error.path”: “/data/error”,“input.file.pattern”: “.*\.csv”,“schema.generation.enabled”:“true”,“csv.first.row.as.header”:“true”}’

This error appears:

HTTP/1.1 415 Unsupported Media Type
Date: Wed, 18 Jan 2023 10:25:39 GMT
Content-Length: 62
Server: Jetty(9.4.33.v20201020)

{“error_code”:415,“message”:“HTTP 415 Unsupported Media Type”}curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 69:
.*\.csv,schema.generation.enabled:true,csv.first.row.as.header:true}’

Can anyone help me?

hey @SeverusP

according to @rmoff’s post there is a backslack missing for
"input.file.pattern": ".*\\.csv",

Nevertheless could please share you’re example formatted as “preformatted text” (
image )

cause at the moment the quotation marks and apostrophes look not well formatted guess it’s related to discourse.

best,
michael

Thanks for the answers.

curl -i -X PUT http://localhost:8083/connectors/source-csv-spooldir-00/config -H 'Accept:application/json' -H 'Content-Type:application/json' -d '{"connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector","topic": "orders_spooldir_00","input.path": "/data/unprocessed","finished.path": "/data/processed","error.path": "/data/error","input.file.pattern": ".*\\.csv","schema.generation.enabled":"true","csv.first.row.as.header":"true"}'

I went ahead but now the error is about the media type (on cmd prompt)

HTTP/1.1 415 Unsupported Media Type
Date: Wed, 18 Jan 2023 13:09:27 GMT
Content-Length: 62
Server: Jetty(9.4.33.v20201020)

{"error_code":415,"message":"HTTP 415 Unsupported Media Type"}curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 69:
.*\\.csv,schema.generation.enabled:true,csv.first.row.as.header:true}'

Different error with same command on gitbash

{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value '/data/unprocessed' must be a directory. for configuration input.path\nInvalid value '/data/error' must be a directory. for configuration error.path\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}

hi @SeverusP

just tested by myself.
works for me if the directories for input.path and so have been created before issuing the command above.

I assume the directories don’t exist in your current setup?

best,
michael

I’ve downloaded the github repo on my desktop

paths:

C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\data\error
C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\data\processed
C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\data\unprocessed

ok I see
so you’re using the following compose:

not sure whether it might be related to Windows access problems or insufficient privs?
what does docker logs for the kafka-connect container say?

best,
michael

I don’t know… i’m trying to practice Kafka with my company pc, maybe I’ve problem related to the priveleges.

Anyway…this is the log, same issue:

ERROR Uncaught exception in REST call to /connectors/source-csv-spooldir-00/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
2023-01-18 16:11:06 javax.ws.rs.NotSupportedException: HTTP 415 Unsupported Media Type
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.getMethodRouter(MethodSelectingRouter.java:412)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.access$000(MethodSelectingRouter.java:73)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.MethodSelectingRouter$4.apply(MethodSelectingRouter.java:665)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.apply(MethodSelectingRouter.java:305)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
2023-01-18 16:11:06     at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
2023-01-18 16:11:06     at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
2023-01-18 16:11:06     at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245)
2023-01-18 16:11:06     at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
2023-01-18 16:11:06     at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
2023-01-18 16:11:06     at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
2023-01-18 16:11:06     at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
2023-01-18 16:11:06     at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
2023-01-18 16:11:06     at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
2023-01-18 16:11:06     at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
2023-01-18 16:11:06     at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
2023-01-18 16:11:06     at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
2023-01-18 16:11:06     at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
2023-01-18 16:11:06     at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
2023-01-18 16:11:06     at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
2023-01-18 16:11:06     at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
2023-01-18 16:11:06     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
2023-01-18 16:11:06     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:563)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
2023-01-18 16:11:06     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
2023-01-18 16:11:06     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
2023-01-18 16:11:06     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
2023-01-18 16:11:06     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
2023-01-18 16:11:06     at org.eclipse.jetty.server.Server.handle(Server.java:516)
2023-01-18 16:11:06     at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
2023-01-18 16:11:06     at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
2023-01-18 16:11:06     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
2023-01-18 16:11:06     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
2023-01-18 16:11:06     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
2023-01-18 16:11:06     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
2023-01-18 16:11:06     at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
2023-01-18 16:11:06     at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
2023-01-18 16:11:06     at java.base/java.lang.Thread.run(Thread.java:834)

I see
thanks for sharing the logs

3 things to try:

  1. change the path in docker-compose.yml to a full qualified path

in line 103
change from

    volumes:
      - $PWD/data:/data

to

    volumes:
      - C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\data:/data
  1. change the local directories to something else, eg.
mkdir C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\error
mkdir C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\processed
mkdir C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\unprocessed

adapt the directory in docker-compose.yml

  1. maybe unpopular option

switch to a linux vm :wink:

best,
michael

1 Like

Hi!

Thank you very much for your availability and your advice.

I tried with your volumes configuration in compose file.

volumes:
      - C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata

In connector’s JSON I set:

"input.path": "C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\unprocessed"
"finished.path": "C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\processed"
"error.path": "C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\mydata\error"

Unfortunately the error is still the same:

{"error_code":415,"message":"HTTP 415 Unsupported Media Type"}curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 69:
.*\\.csv,schema.generation.enabled:true,csv.first.row.as.header:true}'

Hi @SeverusP

no worries :slight_smile:

There is a slight issue in your config:

input.path, finished.path and error.path have to be a directory inside the docker container
so this stays the same as in your original curl command.

volume config in your compose file is wrong (or shortened by discourse), should be

after my reading my post above it seems a little misleading

best,
michael

Sorry for my beginner mistakes.

I tried another time with these configurations.

JSON:
{"connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector", "topic": "orders_spooldir_00", "input.path": "data\unprocessed", "finished.path": "data\processed", "error.path": "data\error", "input.file.pattern":".*\\.csv","schema.generation.enabled":"true","csv.first.row.as.header":"true"}

my docker-compose

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.1.0
    container_name: broker
    depends_on:
      - zookeeper
    ports:
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    # An important note about accessing Kafka from clients on other machines:
    # -----------------------------------------------------------------------
    #
    # The config used here exposes port 9092 for _external_ connections to the broker
    # i.e. those from _outside_ the docker network. This could be from the host machine
    # running docker, or maybe further afield if you've got a more complicated setup.
    # If the latter is true, you will need to change the value 'localhost' in
    # KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those
    # remote clients
    #
    # For connections _internal_ to the docker network, such as from other services
    # and components, use broker:29092.
    #
    # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    #
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

  schema-registry:
    image: confluentinc/cp-schema-registry:6.1.0
    container_name: schema-registry
    ports: 
      - 8081:8081
    depends_on:
      - zookeeper
      - broker
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:29092

  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.1.0
    container_name: kafka-connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/'
    command: 
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.0.2
        confluent-hub install --no-prompt jcustenborder/kafka-connect-spooldir:2.0.60
        confluent-hub install --no-prompt streamthoughts/kafka-connect-file-pulse:1.5.0
        #
        # -----------
        # Launch the Kafka Connect worker
        /etc/confluent/docker/run &
        #
        # Don't exit
        sleep infinity
    volumes:
      - C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka\data:/data

My directory in C:\Users\dinardo\Desktop\demo-scene-master\csv-to-kafka:

.idea
data(with inside error, processed, unprocessed)
kafka-connect-spooldire
docker-compose

looks promising error is still the same?

Yes, unfortunatly.

Thank you!!

ok :expressionless:

one thing, there is a missing leading “/” for input.path, finished.path and error.path:

input.path": "data\unprocessed", "finished.path": "data\processed", "error.path": "data\error"

should be
input.path": "/data/unprocessed", "finished.path": "/data/processed", "error.path": "/data/error"

1 Like

Always the same error.

I don’t really know but it seems that something is wrong with this settings:

-H 'Accept:application/json'
-H 'Content-Type:application/json'

hmm

just tested the demo on my local machine (mac os) and it works for me :face_with_hand_over_mouth:

one thing to try:
remove the volume mount for the kafka-connect container and see what happens then .
just to be sure that windows is not doing any strange things

here’s a small example just pushed to git

best,
michael

Hi! Thank you for your time

Still not working for me.

The strange think is that, if I try the command on GitBash instead of cmd the error is different

{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value '/data/unprocessed' must be a directory. for configuration input.path\nInvalid value '/data/error' must be a directory. for configuration error.path\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}

so next try :wink:

  1. start the env with docker-compose (once again without the volume mount)

  2. connect to the kafka connect container and create the dirs needed

  3. execute the curl command

1 Like

How I did it:

$ winpty docker exec -it kafka-connect bash
[appuser@ff0799e74189 ~]$ mkdir data
[appuser@ff0799e74189 ~]$ ls
data
[appuser@ff0799e74189 ~]$ cd data
[appuser@ff0799e74189 data]$ mkdir error
[appuser@ff0799e74189 data]$ mkdir processed
[appuser@ff0799e74189 data]$ mkdir unprocessed
[appuser@ff0799e74189 data]$ ls
error  processed  unprocessed

Command:

curl.exe -i -X PUT http://localhost:8083/connectors/source-csv-spooldir-00/config -H 'Accept:application/json' -H 'Content-Type:application/json' -d '{"connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector","topic": "orders_spooldir_00","input.path": "/data/unprocessed","finished.path": "/data/processed","error.path": "/data/error","input.file.pattern": ".*\\.csv","schema.generation.enabled":"true","csv.first.row.as.header":"true"}'

Same error.

{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value '/data/unprocessed' must be a directory. for configuration input.path\nInvalid value '/data/error' must be a directory. for configuration error.path\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}

mmh wrong directory

should be
mkdir /data/

or set the dir in the curl command to
/home/appuser/data/...

best,
michael