Elasticsearch Sink connector with amazon cognito

Kafka Connect
#1

I am getting errors when trying to implement amazon cognito authentication with the Elasticsearch connector. Has anyone tested this out?

From what I see in the doc I used
connection.username and connection.password

Will this work? Any insight will be great.

#2

What errors are you getting?

Can you share your connector config & version details?

#3

Let me re run it. I disabled the cognito to make sure it was working again.
The first error I saw before I added the connection.username was that there was an error authenticating which is understood.

After I added the below config, this was one of the errors I saw.

Error: Failed to chec$ org.apache.kafka.connect.errors.ConnectException: Failed to check if index prod-parser-logs-2021-03-24 exists.

Here is the connector config.

  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "_doc",
  "tasks.max": "2",
  "topics": "prod-parser-logs",
  "transforms.routeTS.type": "org.apache.kafka.connect.transforms.TimestampRouter",
  "transforms": "routeTS",
  "transforms.routeTS.topic.format": "${topic}-${timestamp}",
  "connection.username":"test",
  "connection.password":"%test%",
  "key.ignore": "true",
  "schema.ignore": "true",
  "transforms.routeTS.timestamp.format": "YYYY-MM-dd",
  "value.converter.schemas.enable": "false",
  "name": "elastic-search-prod-parser-logs2",
  "connection.url": "https://****",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter"
}

Version of ES I am using is 7.7.
Confluent version I am using 6.0.

Let me know if you want me to test it again and see if there are other errors I missed.

Thank you

#4

I tested this on another brand new instance but seem to get the same errors.
Do you know if this works with amazon cognito authentication? Any other Ideas I can try?

I am seeing this ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 403 Forbidden]]
Just seems like an authentication issues.

#5

So it looks like this will not work unless the requests are signed by AWS. I did not see any provision to sign requests to Elasticsearch. Do you know if this is supported?

#6

AFAIK this is not supported.

#7

Understand. Thank you.