Enormous amount of TIME_WAIT connections with non-buffered (direct) output

[TaLoN1x]

I am using this configuration:

<match **>
  @type kafka
  brokers somebrokers:9092
  default_topic 'sometopic'
  output_data_type json
  max_send_retries 300
  required_acks 0
</match>

And with this I do have a lot of TIME_WAIT connections to kafka node:

# netstat -tulnap | grep TIME_WAIT | wc -l
19699

# netstat -tulnap | grep TIME_WAIT
...
tcp        0      0 192.168.100.22:41907     192.168.100.55:9092      TIME_WAIT   -
tcp        0      0 192.168.100.22:43715     192.168.100.55:9092      TIME_WAIT   -
tcp        0      0 192.168.100.22:35498     192.168.100.55:9092      TIME_WAIT   -
tcp        0      0 192.168.100.22:56291     192.168.100.55:9092      TIME_WAIT   -
tcp        0      0 192.168.100.22:53716     192.168.100.55:9092      TIME_WAIT   -

What could be the cause of fluentd keeping those connections alive until timeout? As fas as I've managed to debug fluentd is not reusing any of established connection in this mode as well.... With this is produces unreasonable load on network devices.

[repeatedly]

What could be the cause of fluentd keeping those connections alive until timeout?

fluent-plugin-kafka uses ruby-kafka library internally so it depends on ruby-kafka's setting.
out_kafka plugin is for testing purporse without buffering, 1 connection per 1 event in most cases, so I'm not sure this problem happens with buffer based out_kafka_buffered or out_kafka2 plugins.

[TaLoN1x]

with buffered connection it doesn't happen in such scale, but still handles connection closure in the wrong way. With buffered output there is a problem of buffer flush, as describes in here:
#101

I am trying to understand what configuration is most relevant for the use case, when I don't need to handle 100% messages and can accept drops, but without building up a buffer trail (fluentd not being able to handle buffer flushes and starts growing up the buffer) not bunch of TCP_WAITS.

Can u advise?

[repeatedly]

Can u advise?

What the advise do you want?
How to reduce TCP_WAITS or other?

[TaLoN1x]

Ok, I've tested out both buffered and non buffered options. To my surprise it had mostly the same result...

For the buffered output I used this configuration:

<match **>
    @type kafka2

    brokers             somebrokers:9092 # Set brokers directly

    default_topic sometopic
    <format>
      @type json
    </format>

    <buffer topic,time>
        timekey 10s
        timekey_wait 5s
        flush_at_shutdown true
        retry_wait 5s
        flush_mode immediate
        flush_thread_count 14
        overflow_action throw_exception
    </buffer>

    # ruby-kafka producer options
    max_send_retries 300
    required_acks 0
    max_bytes 1000000
    kafka_agg_max_bytes 1000000
    max_send_limit_bytes 1000000
    discard_kafka_delivery_failed false
    buffer_chunk_limit 1m

  </match>

It still falls to the same problem:

# netstat -tulnap | grep TIME_WAIT | wc -l
16384

I've gone deeper and inspected traffic...
The root cause is that while connection is being closed, Kafka sends FIN_WAIT2 and expects Fluentd to answer with FIN_ACK, but this never happens...

What makes it even more interesting, is that doing all the same with just ruby-kafka I could not replicate the issue.

[TaLoN1x]

just for reference. old one kafka_buffered worked just fine!

[repeatedly]

Interesting. out_kafka2 and out_kafka_buffered uses same approches and behaviour should be same.
I will check out_kafka2 code.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days