hi everyone, i am having empty rows in csv , i am using spooldir connector i want this empty data to be replaced by a value when transferred from csv to kafka or when transferred from kafka to cassandra
note : i am using datastax connector from kafka to cassanda , also i tried to set default but nothing happened
thanks in advance
It sounds like you’re reading in blank lines from your source CSV files? If this is correct, Filter (Apache Kafka) | Confluent Documentation is probably the easiest solution. It allows you to filter out things like blank lines. There are other Kafka Connect SMTs that come with Confluent packaging or Apache Kafka, they are linked in the doc above.
1 Like
hi mitchel yes iam reading blank lines from csv so i want to replace them by other value , i donot want to drop those nulls ?
Then something like TombstoneHandler | Confluent Documentation or writing your own would be appropriate.
1 Like
I tried to write my own but you know Custom transformations are not available for managed connectors
With hosted Confluent Connect you really don’t have any other options except to either fix the files before the ingested into Kafka. Or ingest them into topic-a, stream process them to fix whatever(ksql is a great option here), then produce them to topic-b for consumption.
2 Likes
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.