Hello,
I got a problem with something that looks like very basic functionality of KSQL.
I’m creating a table as select, from the stream:
CREATE OR REPLACE TABLE t1
WITH (KAFKA_TOPIC='pipeline1', PARTITIONS=12, REPLICAS=3, TIMESTAMP='timestamp') AS SELECT
id1,
latest_by_offset(v1),
latest_by_offset(v2)
FROM s1
GROUP by id1
EMIT CHANGES;
In stream s1, I’ve got 3 records for key (id) 8 with offset 0, 1, 2
0 is creation, 1 is update, 2 is tombstone record with value null
However I still see that record in KSQL table (it wasn’t removed). I don’t see tombstone record in topic ‘pipeline1’. How to enable propagation of tombstone record to downstream tables?
PS. I’m using ksqldb Version 0.20.0
Because your input s1 is a STREAM there is no tombstone semantics… The aggregation will treat the <key,null> record as invalid and drop it (an STREAM-aggregation cannot semantically “forward/process” a delete because the input record is no delete to begin with).
If you want to materialize a TABLE, you should instead create two TABLES:
CREATE TABLE myTable <schema> WITH (...);
CREATE TALBE materializedTable AS SELECT * FROM myTable;
This way, tombstone semantics are preserved.
Btw: we are currently working on a new feature (KLIP-49) that allows you to do the same in a single statement:
CREATE SOURCE TABLE materializedTable <schema> WITH (...);