For a Kafka message which looks like below:
“messageId”: “my-message-id”,
“senderReferenceNumber”: “my-source-nbr”,
“account”: {
“accountNumber”: “my-account-nbr”
}
When I tried configuring sink connector to get all records with field account.accountnumber in it, which is a nested field with below configuration, data is not getting populated on to elasticsearch but when I tried with senderReferenceNumber, which is not nested then the data is getting moved properly. I used RelaceField SMT here. Not sure why nested field does not work here like account.accountnumber in the below example:
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
type.name=_doc
transforms.selectFields.type=org.apache.kafka.connect.transforms.ReplaceField$Value
connection.password=my password
transforms=selectFields
schema.registry.url=https://my-schem-registry-url/
schema.registry.basic.auth.credentials.source=USER_INFO
elastic.https.ssl.truststore.password=mypassword
value.converter=io.confluent.connect.avro.AvroConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
elastic.https.ssl.truststore.type=JKS
topics=my-kafka-topic
connection.username=el-search-username
value.converter.schema.registry.basic.auth.credentials.source=USER_INFO
transforms.selectFields.include=account.accountNumber
elastic.https.ssl.protocol=TLS
elastic.security.protocol=SSL
connection.url=https://elastic-search-url:9200
key.converter.schema.registry.basic.auth.credentials.source=USER_INFO