Confluent Kafka SMT Filter Problem

Kafka Connect Self-Managed Connectors
5

Hi @rmoff, thank you again for your solution. Finally, we solved the problem.

We removed .value.payload from filter condition and changed the filter like:
"transforms.ignoreNonSuppliers.filter.condition": "$[?(@._class != 'CityDistrict')]",

Because, transforms.Filter$Value is at Value’s level. So, value.payload is not necessary.

Also, we changed filter type like:
"transforms.ignoreNonSuppliers.filter.type": "include",

In the last situation, our config is

{
	"name": "custom-couchbase-connector",
	"config": {
		"name": "custom-couchbase-connector",
		"connector.class": "com.couchbase.connect.kafka.CouchbaseSourceConnector",
		"tasks.max": "1",
		"key.converter": "org.apache.kafka.connect.storage.StringConverter",
		"value.converter": "org.apache.kafka.connect.json.JsonConverter",
		"transforms": "ignoreDeletes,deserializeJson,ignoreNonSuppliers",
		"transforms.ignoreDeletes.type": "com.couchbase.connect.kafka.transform.DropIfNullValue",
		"transforms.ignoreNonSuppliers.type": "io.confluent.connect.transforms.Filter$Value",
		"transforms.ignoreNonSuppliers.filter.condition": "$[?(@._class != 'CityDistrict')]",
		"transforms.ignoreNonSuppliers.filter.type": "include",
		"transforms.ignoreNonSuppliers.missing.or.null.behavior": "exclude",
		"transforms.deserializeJson.type": "com.couchbase.connect.kafka.transform.DeserializeJson",
		"couchbase.seed.nodes": "...",
		"couchbase.username": "...",
		"couchbase.password": "...",
		"couchbase.bucket": "cargo-bucket",
		"couchbase.bootstrap.timeout": "10s",
		"couchbase.source.handler": "com.couchbase.connect.kafka.handler.source.RawJsonSourceHandler",
		"couchbase.event.filter": "com.couchbase.connect.kafka.filter.AllPassFilter",
		"couchbase.stream.from": "SAVED_OFFSET_OR_NOW",
		"couchbase.topic": "my-topic",
		"couchbase.compression": "ENABLED",
		"couchbase.persistence.polling.interval": "0",
		"couchbase.flow.control.buffer": "128m"
	}
}

Thanks to Confluent for help again :slightly_smiling_face:

2 Likes