We are facing an issue with Messages being dropped while consuming into BigQuery.
We use a self-managed BigQuery sink connector (confluent-7.1.1 libraries) pointing to Confluent Cloud.
We use the Java-based Kafka Producer application to produce the messages and the BigQuery sink connector to ingest the data into Google BigQuery.
From our Kafka producer application, we produce billions of messages to Kafka, but all messages are not ingested to BigQuery. A few thousand to millions of messages are being dropped.
Could you help us in resolving the issue? Let us know
- How and where can we see such errors? Does the Kafka or BigQuery sink connector log about these un-ingested messages?
- Any connect worker properties to be changed to resolve the issue.