Elasticsearch Sink connector with amazon cognito

I am getting errors when trying to implement amazon cognito authentication with the Elasticsearch connector. Has anyone tested this out?

From what I see in the doc I used
connection.username and connection.password

Will this work? Any insight will be great.

What errors are you getting?

Can you share your connector config & version details?

Let me re run it. I disabled the cognito to make sure it was working again.
The first error I saw before I added the connection.username was that there was an error authenticating which is understood.

After I added the below config, this was one of the errors I saw.

Error: Failed to chec$ org.apache.kafka.connect.errors.ConnectException: Failed to check if index prod-parser-logs-2021-03-24 exists.

Here is the connector config.

  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "_doc",
  "tasks.max": "2",
  "topics": "prod-parser-logs",
  "transforms.routeTS.type": "org.apache.kafka.connect.transforms.TimestampRouter",
  "transforms": "routeTS",
  "transforms.routeTS.topic.format": "${topic}-${timestamp}",
  "key.ignore": "true",
  "schema.ignore": "true",
  "transforms.routeTS.timestamp.format": "YYYY-MM-dd",
  "value.converter.schemas.enable": "false",
  "name": "elastic-search-prod-parser-logs2",
  "connection.url": "https://****",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter"

Version of ES I am using is 7.7.
Confluent version I am using 6.0.

Let me know if you want me to test it again and see if there are other errors I missed.

Thank you

I tested this on another brand new instance but seem to get the same errors.
Do you know if this works with amazon cognito authentication? Any other Ideas I can try?

I am seeing this ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 403 Forbidden]]
Just seems like an authentication issues.

So it looks like this will not work unless the requests are signed by AWS. I did not see any provision to sign requests to Elasticsearch. Do you know if this is supported?

AFAIK this is not supported.

Understand. Thank you.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.