Hi, we are planning to use Confluent Cloud Elasticsearch connector with our dedicated cluster. When we create the connector, it starts reading data from the beginning offset of the topic. Does Elasticsearch Sink connector supporting reading data from the latest current onwards, instead of start ingesting all the historical data from the topic?
I tried setting “consumer.auto.offset.reset” and “consumer.override.auto.offset.reset” to “latest” but that doesn’t seem to work. Tried resetting the offsets while pausing connector but that wouldn’t work as the consumer group’s state is stable which doesn’t allow resetting the offsets.
Appreciate if anyone has any suggestions on achieving this and if this is possible at all.
Thanks in advance.