When Should You Use a Sink Connector vs Direct Database Writes?

Kafka Connect
1

Hello everyone,

What are the best practices for deciding between:

  • writing directly to the data source from a microservice

  • publishing to Kafka and persisting via a sink connector?

When would you recommend one approach over the other?

2

Some reasons to consider Kafka en route to the external system, roughly in order of decreasing commonness IMO:

  1. Spike resiliency / back pressure. If the microservice is subject to bursty loads that make direct writes to the external system risky (external system isn’t sized to handle the writes) or complex (e.g., throttling), Kafka can serve as a buffer between the the microservice and sink system
  2. Resiliency with respect to sink system failures / downtime
  3. Take advantage of connector’s implementation / properties. This would be connector-dependent, but e.g., some connectors handle sticky external sink client scenarios for you or give you desirable properties like idempotent writes
  4. Stream processing, if applicable. Kafka can be used to process / transform the stream by way of a stream processing framework or within Kafka Connect, e.g., SMTs
  5. Potential for other Kafka consumers / realtime use cases, if applicable

Consider direct writes if these aren’t applicable or don’t seem valuable enough (particularly weighing 1 and 2) to justify adding a box and arrows to the architecture.

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.