Salesforce Source Connector and compacted topics

Kafka Connect Managed Connectors
1

I figured it might be useful to others to write up our experience of getting the Salesforce Source connector talking to a compacted topic.

We set out to achieve the following goals:

  • Expose 10 fields from the SFDC Account object out to a Kafka topic whenever the object is created/updated in SFDC.

  • Set up the topic as a compacted topic so that it can be both treated like a lookup table of current values on an account object as well as providing the ability to turn on “infinite” retention time so that it can become the authoritative source on SFDC account data for those systems that cannot (or do not want to) make direct calls into SFDC itself.

We first looked at the Salesforce Source CDC connector and quickly found that it was limited in functionality. Specifically, it could hook up to standard channels, but the messages produced only contained data for the fields actually changed during a change event. We needed the message to contain all of the field data we are interested in regardless of whether it was changed by the specific event or not. Creating a “custom channel” in SFDC would be how we could have the message contain data for unchanged fields, but the Kafka CDC connector does not support custom channels.

Where we ended up was using the Salesforce Platform Events connector. Using this kafka connector in concert with defining a custom Platform Event within SFDC was the golden ticket for us. We could specify all the fields that we wanted to be part of the message from within SFDC and then leverage the Kafka Platform Events connector to make the communication happen.

The last step was to get the “key” in the kafka message representing the SFDC Id of the account object. Using the “ValueToKey” transform to elevate one of the fields in the message body to be the key was all it took. Now that we have messages keyed by the SFDC Id field, we can leverage partitioning for horizontal scaling while providing the ability to turn the topic into a compacted topic.

Hopefully this helps the next person who is faced with a similar challenge.

2 Likes
2

Hey Mike, thanks for taking the time to write this up and share your findings with the community :+1:

3

Hi Mike, great post. I was wondering how would you send a delete message (null) for a specific Key.

Is that in your workflow or I am missing the point because if you need messages for each create/update for a Salesforce object, why not use PushTopic source connector?

1 Like