Kafka Data loss

Kafka Streams
1

Hello,

We are facing issue in Kafka messages streaming if connectivity break down.
Data is getting lost if connectivity break down or broker is not available.

2

Is this a Kafka Streams app, or actually a producer app?

What’s your topic / client configs? With retries / timeout / replication / acks configured correctly, no data loss should happen. Also, what broker/client versions are you using (newer versions ship with different default configs than older version).

3

we are using producer on one server ‘x’ and consumer on other server says ‘y’.We stopped kafka service on consumer ‘y’.We are sending data from x to y . Few messages are visible after starting kafka on ‘‘y’’ other messages or data is getting lost.
We have added below properties as:
if (rd_kafka_conf_set(conf, “enable.idempotence”, “true”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}
if (rd_kafka_conf_set(conf, “message.send.max.retries”, “200”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}
if (rd_kafka_conf_set(conf, “retry-backoff-ms”, “2000”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}
if (rd_kafka_conf_set(conf, “delivery.timeout.ms”, “150000”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}
if (rd_kafka_conf_set(conf, “acks”, “all”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}
if (rd_kafka_conf_set(conf, “max.in.flight.requests.per.connection”, “1”, errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)
{
TRACE_SHOW(errstr);
}

But getting errstr=No such configuration property: "delivery.timeout.ms" for each property.
Please suggest how can we reloved it.
4

We have checked version with below api:
rd_kafka_version_str()
kafka version:0.11.6
How to check broker version can you please share command for this?

5

So it’s not a Kafka Streams question.

Might be best to re-post in Non-Java Clients - Confluent Community

And there is no kafka version:0.11.6 – seems you are referring to your librdkafka version?

(Also Kafka 0.11. would be very old…)

How to check broker version can you please share command for this?

Brokers don’t have an API to query their version AFAIK. If you are not running the brokers yourself, you would need to ask the team running the brokers what version they deployed.

6

Then how to check kafka version plz suggest