Fluentd not pushing latest logs from a specific namespace in Kubernetes

[taimoor-mirza29]

Describe the bug

Hi,

I've deployed the EFK stack on my bare metal kubernetes cluster. Initially, Fluentd works fine and continues to push the logs of 1 of K8s's namespace (ivr-qa) to ES. However after ~2 hours, it stops. Here's the screenshot

Upon inspecting the logs of Fluentd's pods, I get the following warning:

fluentd-pnppg fluentd 2024-01-15 13:42:01 +0000 [warn]: #0 failed to flush the buffer. retry_time=15 next_retry_seconds=2024-01-15 13:42:29 +0000 chunk="60ef933d4304c5c70d0c9ecefc7ac58f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.kube-logging.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"elastic\", :password=>\"obfuscated\"}): read timeout reached"
fluentd-pnppg fluentd   2024-01-15 13:42:01 +0000 [warn]: #0 suppressed same stacktrace

I don't understand why it says read timeout reached. If I search for ES's own logs in Kibana, I get the latest data every time:

Can someone help me on this? I've spend multiple hours on this already.

Thanks.

To Reproduce

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-logging
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - namespaces
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd # if RBAC is enabled
      serviceAccountName: fluentd # if RBAC is enabled
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1
          env:
            - name:  FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.kube-logging.svc.cluster.local"
            - name:  FLUENT_ELASTICSEARCH_PORT
              value: "9200"
            - name: FLUENT_ELASTICSEARCH_SCHEME
              value: "http"
            - name: FLUENT_ELASTICSEARCH_USER # even if not used they are necessary
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-pv-elastic
                  key: USER_NAME
            - name: FLUENT_ELASTICSEARCH_PASSWORD # even if not used they are necessary
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-pv-elastic
                  key: PASSWORD
          resources:
            limits:
              cpu: 2
              memory: 4Gi
            requests:
              cpu: 1
              memory: 2Gi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: fluentd-config
              mountPath: /fluentd/etc # path of fluentd config file
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: fluentd-config
          configMap:
            name: fluentd-config # name of the config map we will create
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
      kubernetes.io/cluster-service: "true"

Expected behavior

Fluentd should be sending latest logs to Elastic search.

Your Environment

- FluentD => fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1
- ElasticSearch => docker.elastic.co/elasticsearch/elasticsearch:7.17.0
- K8s: v1.22

Your Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: kube-logging
data:
  fluent.conf: |
    <match fluent.**>
        # this tells fluentd to not output its log on stdout
        @type null
    </match>
    # here we read the logs from Docker's containers and parse them
    <source>
      @type tail      
      path /var/log/containers/*ivr-uat*.log,/var/log/containers/*keycloak*.log
      exclude_path ["/var/log/containers/*istio*.log"]
      pos_file /var/log/app.log.pos      
      tag kubernetes.*
      read_from_head true
      <parse>
        @type none
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>    
    # we use kubernetes metadata plugin to add metadatas to the log
    <filter kubernetes.**>
        @type kubernetes_metadata
    </filter>
    # we send the logs to Elasticsearch
    <match **>
       @type elasticsearch_dynamic
       @log_level info
       include_tag_key true
       host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
       port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
       user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
       password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
       scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
       ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
       reload_connections true
       logstash_prefix ivr-uat
       logstash_format true
       <buffer>
           @type file
           path /var/log/fluentd-buffers/kubernetes.system.buffer
           flush_mode interval
           retry_type exponential_backoff
           flush_thread_count 2
           flush_interval 30s
           retry_forever true
           retry_max_interval 60
           chunk_limit_size 20M
           queue_limit_length 32
           overflow_action block
       </buffer>
    </match>
---

Your Error Log

fluentd-pnppg fluentd 2024-01-15 13:42:01 +0000 [warn]: #0 failed to flush the buffer. retry_time=15 next_retry_seconds=2024-01-15 13:42:29 +0000 chunk="60ef933d4304c5c70d0c9ecefc7ac58f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.kube-logging.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"elastic\", :password=>\"obfuscated\"}): read timeout reached"
fluentd-pnppg fluentd   2024-01-15 13:42:01 +0000 [warn]: #0 suppressed same stacktrace

### Additional context

_No response_

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days