# Fluentd not consuming from kafka

**URL:** <https://forum.confluent.io/t/fluentd-not-consuming-from-kafka/2637>\
**Category:** Kafka Connect\
**Created:** [2 September 2021 11:00 UTC](https://forum.confluent.io/t/fluentd-not-consuming-from-kafka/2637 "2021-09-02T11:00:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MBmousavi](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mbmousavi/32/1043_2.png) [@MBmousavi](https://forum.confluent.io/u/MBmousavi)\
**Post date:** [2 September 2021 11:00 UTC](https://forum.confluent.io/t/fluentd-not-consuming-from-kafka/2637/1 "2021-09-02T11:00:40Z")

</div>

Hi, I have a EFK stack on k8s. There is a dumy app that generates logs and I collect the logs via fluent Bit and put in on a single node kafka, on the other side there is a deamon set of fluentd that consume the logs from the topic and insert it on elasticsearch. so far everything is fine. I came up with a scenario that fluentd is crashed. when I delete the replicaset from k8s and create it again after a while, I expect that fluentd starts to consume the logs from topic where it left it. But it ignore it and start to insert new logs to elasticseach and a now there is a gap in elasticseach. I checked the `time` on logs but the gap is not in them.

 ![1](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/b/b0ca412ba3c505b6546f4eb4c41648bf257cf067.png)

This is my config files:  
fluent-bit:

```auto
    [INPUT]
        Name tail
        Tag docker-log-gen.*
        Path /var/log/containers/docker-log-generator*.log
        Parser docker
        DB /var/log/flb_kube.db
        Mem_Buf_Limit 100MB
        Skip_Long_Lines On
        Refresh_Interval 10

    [OUTPUT]
        Name kafka
        Match docker-log-gen.*
        Brokers kafka-service:9092
        Logstash_Format Off
        topics fluentbit-docker-log-gen
        Replace_Dots On
        Retry_Limit False

```

fluentd:

```auto
<source>
   @type kafka
   @log_level info
   brokers kafka-service
   topics fluentbit-docker-log-gen
   time_key time
</source>
<match *>
   @type elasticsearch
   time_key time
   ....
</match>

```

I will be very appreciate if you can help with that.  
Thanx a lot

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [2 September 2021 13:17 UTC](https://forum.confluent.io/t/fluentd-not-consuming-from-kafka/2637/2 "2021-09-02T13:17:17Z")

</div>

This sounds like more of a Fluentd question than something the Confluent community can necessarily help with.

Have you considered using [Kafka Connect to stream the data to Elasticsearch](https://www.youtube.com/watch?v=Cq-2eGxOCc8&list=PL5T99fPsK7ppB_AbZhBhTyKHtHWZLWIJ8&index=7) instead? I’m not familiar with fluentd so perhaps this suggestion is redundant, but Kafka Connect supports delivery guarentees, restarts, etc etc so it may be useful here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [2 October 2021 13:17 UTC](https://forum.confluent.io/t/fluentd-not-consuming-from-kafka/2637/3 "2021-10-02T13:17:55Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.