# Amazon SQS source connector ValueToKey;extractKey **URL:** \ **Category:** Kafka Connect\ **Created:** [6 June 2022 16:59 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218 "2022-06-06T16:59:59Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [6 June 2022 16:59 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/1 "2022-06-06T16:59:59Z")
Hi All, I am looking for a way to extract a key from SQS source connector value record schema which is string. Can I use JSON PATH to achieve that ? ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [9 June 2022 12:34 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/2 "2022-06-09T12:34:30Z")
Should be possible, yes, but there’s no builtin transform for json path. You could also use ksql, for example, to parse your string to do the same ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [10 June 2022 15:23 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/3 "2022-06-10T15:23:27Z")
Thank you for your response. I am working with confluent team to give me an example of using ksql. ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [10 June 2022 21:02 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/4 "2022-06-10T21:02:36Z")
You can find some examples at [Working with nested JSON using ksqlDB](https://developer.confluent.io/tutorials/working-with-nested-json/ksql.html) ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [10 June 2022 22:10 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/5 "2022-06-10T22:10:13Z")
Thank you for the response. Does this mean I need to first load SQS to single partition topic (to maintain order) and then use ksqlDB to create a key for another topic with multiple partitions (for scaling) to consume from ? ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [10 June 2022 22:23 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/6 "2022-06-10T22:23:28Z")
Ksql is just one option, but yes, you’ll need consume, parse, then to write to another topic with a new key in order to scale the consumption. Another option would be a Kafka producer inside a lambda triggered by sqs ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [11 June 2022 02:36 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/7 "2022-06-11T02:36:50Z")
So I think I can’t horizontally scale SQS Source connector even using ksqlDB. Connector gives great advantage like at least once delivery and keeping orders using key. If I use lambda and write my own producer can I achieve at least once delivery at scale ? ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [11 June 2022 02:50 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/8 "2022-06-11T02:50:04Z")
> cannot scale… even using ksql You’d use ksql (or any other consumer, as mentioned) to write to a new, more distributed Kafka topic. Of course, you’d then lose ordering. Source connector doesn’t really matter. * * * > connector gives great advantage like at least once delivery I’m not completely familiar with SQS, but I suspect you’d get **at most** once delivery. For example, if the lambda or connector consumed the event, but your producer failed to forward the record to Kafka, that data is gone. But yes, the full event details are parsable, so you can set the keys and/or partition details to whatever you want. You could even send events to multiple topics at once. ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [11 June 2022 03:55 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/9 "2022-06-11T03:55:02Z")
Actually the connector guarantees at least once delivery and multiple tasks. Please see here. [https://docs.confluent.io/kafka-connect-sqs/current/overview.html](https://docs.confluent.io/kafka-connect-sqs/current/overview.html) This is hard to build if I use SQS Lambda triggers. Before I jump on trying lambda, do we have a Header To Key SMT or custom SMT support in confluent cloud ? Also, can I use multiple SQS source connector tasks producing records to single partition topic ? ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [11 June 2022 04:33 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/10 "2022-06-11T04:33:39Z")
Yes, my bad, I see SQS documentation says it natively offers at least once delivery… > custom SMT support in confluent cloud ? Not that I know of. > Header To Key SMT [https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/HeaderToField.html](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/HeaderToField.html) > use multiple SQS source connector tasks producing records to single partition topic I cannot think of why that wouldn’t be possible. The connector just acts as a producer and can write from as many queues as needed into one or many Kafka topics. > hard to build if I use SQS Lambda triggers. Don’t think so. You’d just import a relevant Kafka producer client and call send method, just like you would from any other app outside of lambda. Choose a language you’re most familiar with. ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [11 June 2022 13:41 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/11 "2022-06-11T13:41:17Z")
You mean to say I can use multiple tasks do you also mean I can use it source from single SQS queue to single partition topic ? The HeaderToField$Key looks promising. Is it offered in confluent cloud ? Thank you for your help. ---
**Author:** ![OneCricketeer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/onecricketeer/32/1551_2.png) [@OneCricketeer](https://forum.confluent.io/u/OneCricketeer)\ **Post date:** [11 June 2022 17:28 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/12 "2022-06-11T17:28:02Z")
The documentation says the connector supports multiple tasks. That SMT is a custom one. I don’t use Confluent Cloud to know what is available or not. ---
**Author:** ![JP2022](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JP2022](https://forum.confluent.io/u/JP2022)\ **Post date:** [11 June 2022 17:53 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/13 "2022-06-11T17:53:15Z")
Thank you for all you help! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [11 July 2022 17:53 UTC](https://forum.confluent.io/t/amazon-sqs-source-connector-valuetokey-extractkey/5218/14 "2022-07-11T17:53:26Z")
Closed on 11 July 2022 17:53 UTC This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.