The first message from .NET producer takes extremely long time to reach the consumer

Clients Non-Java Clients
1

Hello! I am new to Kafka and am trying to get my first producer client (see the code below) to work.

        var config = new ProducerConfig()
            {
                BootstrapServers = "localhost:9092",
                ClientId = Dns.GetHostName(),
            };

        var producer = new ProducerBuilder<Null, string>(config).Build();
        producer.Produce("activity", new Message<Null, string> {Value = "message"});

The problem I am running into is that it takes a long time (~85 seconds) for the first produced message to show up on the consumer side; the subsequent messages after the first one show up almost immediately.

My dev and test environment is Windows 10 + .NET 4.8 with the following software versions:

kafka_2.13-2.7.0
apache-zookeeper-3.6.2-bin
Confluent.Kafka 1.5.3

Kafka, ZooKeeper and the test application all run on the same Windows 10 machine.

I was using kafka-console-consumer.bat as the consumer. When I used kafka-console-producer.bat as the producer, the text that I entered on the producer side gets picked up immediately by the consumer:

kafka-console-consumer.bat --topic activity --bootstrap-server localhost:9092
kafka-console-producer.bat --topic activity --bootstrap-server localhost:9092

So my question is what could be causing the different behavior between the .NET producer and kafka-console-producer.bat? What parameters on ProducerConfig can I tune to speed up the first message?

2

Hi @wqiu , have you tried checking with kafkacat directly to figure out if the delay is on the consume or produce side?
If you see the message quickly with kafkacat then probably is a config on the consumer side.
Have you played around with linger.ms config by chance?

3

@gianlucanatali Tweaking linger.ms did not help. I have not tried kafkacat yet, but since kafka-console-consumer.bat and kafka-console-producer.bat work fine, it is more likely to a producer side issue.

4

Here’s more information: the issue appears to be related to VPN. The test setup works fine when my laptop is not connected to the company VPN. The first message will be delayed for about 85 seconds if my laptop is on VPN. So it appears Confluent.Kafka might be distracted by the presence of VPN connection.

5

You can get very verbose logging by setting the producers Debug config property to ‘all’. That will help identify specifically what the problem is.

1 Like