# Elasticsearch Sink connector with amazon cognito

**URL:** <https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289>\
**Category:** Kafka Connect\
**Created:** [24 March 2021 13:43 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289 "2021-03-24T13:43:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![robbyd](https://avatars.discourse-cdn.com/v4/letter/r/5e9695/32.png) [@robbyd](https://forum.confluent.io/u/robbyd)\
**Post date:** [24 March 2021 13:43 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/1 "2021-03-24T13:43:39Z")

</div>

I am getting errors when trying to implement amazon cognito authentication with the Elasticsearch connector. Has anyone tested this out?

From what I see in the doc I used  
`connection.username` and `connection.password`

Will this work? Any insight will be great.

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [24 March 2021 13:56 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/2 "2021-03-24T13:56:48Z")

</div>

What errors are you getting?

Can you share your connector config & version details?

---

<div class="post-metadata">

**Author:** ![robbyd](https://avatars.discourse-cdn.com/v4/letter/r/5e9695/32.png) [@robbyd](https://forum.confluent.io/u/robbyd)\
**Post date:** [24 March 2021 14:23 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/3 "2021-03-24T14:23:32Z")

</div>

Let me re run it. I disabled the cognito to make sure it was working again.  
The first error I saw before I added the `connection.username` was that there was an error authenticating which is understood.

After I added the below config, this was one of the errors I saw.

`Error: Failed to chec$ org.apache.kafka.connect.errors.ConnectException: Failed to check if index prod-parser-logs-2021-03-24 exists.`

Here is the connector config.

```auto
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "_doc",
  "tasks.max": "2",
  "topics": "prod-parser-logs",
  "transforms.routeTS.type": "org.apache.kafka.connect.transforms.TimestampRouter",
  "transforms": "routeTS",
  "transforms.routeTS.topic.format": "${topic}-${timestamp}",
  "connection.username":"test",
  "connection.password":"%test%",
  "key.ignore": "true",
  "schema.ignore": "true",
  "transforms.routeTS.timestamp.format": "YYYY-MM-dd",
  "value.converter.schemas.enable": "false",
  "name": "elastic-search-prod-parser-logs2",
  "connection.url": "https:// ****",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter"
}

```

Version of ES I am using is 7.7.  
Confluent version I am using 6.0.

Let me know if you want me to test it again and see if there are other errors I missed.

Thank you

---

<div class="post-metadata">

**Author:** ![robbyd](https://avatars.discourse-cdn.com/v4/letter/r/5e9695/32.png) [@robbyd](https://forum.confluent.io/u/robbyd)\
**Post date:** [29 March 2021 15:55 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/4 "2021-03-29T15:55:23Z")

</div>

I tested this on another brand new instance but seem to get the same errors.  
Do you know if this works with amazon cognito authentication? Any other Ideas I can try?

I am seeing this `ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 403 Forbidden]]`  
Just seems like an authentication issues.

---

<div class="post-metadata">

**Author:** ![robbyd](https://avatars.discourse-cdn.com/v4/letter/r/5e9695/32.png) [@robbyd](https://forum.confluent.io/u/robbyd)\
**Post date:** [30 March 2021 12:46 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/5 "2021-03-30T12:46:44Z")

</div>

So it looks like this will not work unless the requests are signed by AWS. I did not see any provision to sign requests to Elasticsearch. Do you know if this is supported?

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [30 March 2021 13:10 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/6 "2021-03-30T13:10:52Z")

</div>

AFAIK this is not supported.

---

<div class="post-metadata">

**Author:** ![robbyd](https://avatars.discourse-cdn.com/v4/letter/r/5e9695/32.png) [@robbyd](https://forum.confluent.io/u/robbyd)\
**Post date:** [30 March 2021 19:17 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/7 "2021-03-30T19:17:14Z")

</div>

Understand. Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [29 April 2021 19:17 UTC](https://forum.confluent.io/t/elasticsearch-sink-connector-with-amazon-cognito/1289/8 "2021-04-29T19:17:36Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.