Request for Enhanced Context Information in Kafka Connector Operations

[RobertHAtEnova]

I am writing to share a suggestion for enhancing the functionality of Kafka connectors API within Confluent Cloud.

Currently, when we manually or programmatically perform API operations such as START, PAUSE, RESUME, or DELETE on connectors or connector tasks, we lack the ability to add contextual information to these actions. I believe that adding the capability to include miscellaneous context information in the body of PUT or POST requests for these operations would be immensely valuable.

Specifically, I propose the inclusion of context information such as:

1. Who initiated the operation (e.g., a client-side user name or id).
2. When the operation was performed (timestamp).
3. Why the operation was initiated (brief description or reason code).

This additional context information would serve several purposes:

1. Enhanced Auditability: Having a record of who initiated connector operations and when they occurred would facilitate auditing and compliance efforts.
2. Improved Troubleshooting: Contextual information about why certain operations were initiated can aid in troubleshooting and diagnosing issues with connectors.
3. Better Insights: The ability to query the status of connectors and retrieve associated context information would provide valuable insights into the history of connector operations. For example, knowing how long connectors have been in a specific state could be useful for monitoring and performance analysis.

I believe that implementing this feature would greatly enhance the usability and manageability of Kafka connectors within Confluent Cloud, and we hope that you will consider my suggestion. I welcome any feedback on this.

[dtroiano]

Thank you for the suggestion.

@RobertHAtEnova

1. Who initiated the operation (e.g., a client-side user name or id).
2. When the operation was performed (timestamp).
3. Why the operation was initiated (brief description or reason code).

Confluent Cloud audit logs provide 1 and 2. See here for connector event examples. This happens implicitly without having to tack “who” and “when” context onto the request.

The “why” isn’t doable today – I’m thinking that auditable API-based events could have a custom metadata field in the request that passes through to the audit log event. Custom rather than an explicit reason field or codes so that other use cases could piggyback on it, say tracking the person who approved an operation. I’ve written up product feedback to this effect but please let me know if you would adjust it or add anything!