Process only specific properties/fields from the Azure CosmosDB to Kafka topic using CosmosDB-Source-Connector

[saurabh]

Hi Team,

Our requirement is to pull data from the Azure CosmosDB and send it to the Kafka topic only with some specific properties in JSON format using CosmosDB source connect.

We have done the setup as given in the below article: Kafka Connect for Azure Cosmos DB - Source connector | Microsoft Docs

So currently records from the Azure Cosmos DB are published to Kafka topic properly.

Now how we can send only some of the properties to the Kafka topic?

Sample data in the Azure Cosmos DB:

    {
      "Address": "string",
      "Company": "string",
      "Country": "string",
      "Email": "login9@example.com",
      "Login": "login9",
      "Name": "string",
      "Password": "login9#login",
      "Telephone": "string",
      "id": "f1b0bf30-51e7-468e-ac09-39f471490a4a",
    }

Expected Value in Kafka topic:

    {      
      "Email": "login9@example.com",
      "Login": "login9",
      "Name": "string",
      "id": "f1b0bf30-51e7-468e-ac09-39f471490a4a",
    }

Thanks
Saurabh

[abellemare]

Hi @saurabh - Welcome to the community!

Since you’re already using Kafka Connect, I think the next easiest step would be to use Single Message Transforms (SMTs). The docs are for CP, but hold true even if not using CP.

For example, you could mask the values using MaskField | Confluent Documentation

Or, you could remove it entirely using ReplaceField | Confluent Documentation

Here’s a blog post that talks about SMTs in more detail: Single Message Transforms in Kafka Connect

If you’re managing your own connectors and Kafka Connect, you’ll need to look into adding the necessary configs to your connector.

Hope this helps!