Kafka-connect-bigquery sink connector failure issue

Kafka Connect
1

Getting below error while creating a new bigquery sink connector, not sure why I am getting this error same is observed for existing connectors also the task remain in the running state but the connector fails with exact same error trace, please help it is a production setup and we are not able to figure-out how to move ahead.

{"name":"sink_connector_name","connector":{"state":"FAILED","worker_id":"ip-172-***_hostname:8083","trace":"java.lang.NullPointerException\n\tat com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.fromPb(StandardTableDefinition.java:121)\n\tat com.google.cloud.bigquery.StandardTableDefinition$Builder.<init>(StandardTableDefinition.java:157)\n\tat com.google.cloud.bigquery.StandardTableDefinition$Builder.<init>(StandardTableDefinition.java:127)\n\tat com.google.cloud.bigquery.StandardTableDefinition.fromPb(StandardTableDefinition.java:326)\n\tat com.google.cloud.bigquery.TableDefinition.fromPb(TableDefinition.java:218)\n\tat com.google.cloud.bigquery.TableInfo$BuilderImpl.<init>(TableInfo.java:162)\n\tat com.google.cloud.bigquery.Table.fromPb(Table.java:559)\n\tat com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:385)\n\tat com.wepay.kafka.connect.bigquery.BigQuerySinkConnector.ensureExistingTables(BigQuerySinkConnector.java:111)\n\tat com.wepay.kafka.connect.bigquery.BigQuerySinkConnector.ensureExistingTables(BigQuerySinkConnector.java:136)\n\tat com.wepay.kafka.connect.bigquery.BigQuerySinkConnector.start(BigQuerySinkConnector.java:155)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:111)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:136)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.transitionTo(WorkerConnector.java:196)\n\tat org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:252)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:1079)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1300(DistributedHerder.java:117)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder$11.call(DistributedHerder.java:797)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder$11.call(DistributedHerder.java:783)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:296)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:245)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"},"tasks":[],"type":"sink"}

we are using confluent-5.3.1-2.12

2

Confluent Platform 5.3 is pretty old (latest is 6.2). My first port of call would be to try a more recent version (particularly since a NullPointerException could be a bug or rough edge that’s since been fixed)

3

@rmoff Thanks for your response !! Got it , but the same is in production from 2019(multiple source connector and lot of sink connectors ) and this issue started now. Also upgrading the platform will take some time as we will have to follow proper process and testing. would you be able to help on our current version ?

4

@Satyaa can you post your configuration (secrets redacted) and include the version of the BigQuery connector you are seeing the exception in?

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.