SMT ReplaceField whitelist and table.whitelist

Kafka Connect
1

Hello,

I’m using a JDBC Source connector with a list of tables (table.whitelist).
Can i use the SMT RemplaceField to filter filters (whitelist or blacklist) with that list of tables ?
if so, what is the correct syntax ? Dot notation ?

    "transforms"   : "selectFields",
    "transforms.selectFields.type"     : "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.selectFields.whitelist": "table1.field1, table1.field2, table2.field3"

if I can’t specify the table’s name in the field whitelist or blacklist, does it means all the field must exist in the all tables ?

Thanks

2

transforms are done after the data has been marshaled into the Struct construct within Connect; I do not believe select field allows for including the sturcture name (usually what is mapped to the table) for the mapping.

What you can do is write a predicate to only apply transforms if a condition has been met.

if each table is going to a separate topic, you can use a predicate of ‘org.apache.kafka.connect.transforms.predicates.TopicNameMatches’ to identify if the correct topic is “table1” or “table2” (or whatever it is mapped to).

Robin has an excellent blog/tutorial on this at 🎄 Twelve Days of SMT 🎄 - Day 11: Predicate and Filter

1 Like
3

Ok, thanks
I’ll try that !

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.