Fluentd not connecting to AWS Elasticsearch Service

[amin224]

I setup an Elasticsearch + Kibana for my cluster and I choose public access for first time use and testing. I am trying to install fluentd in order to centralize my application logs. I researched a lot of other methods like Clock-watch and fluent-bit but I went with fluentd as it looked very simple. My issue is that when i deploy my fluent yaml file it is not connecting to Elasticsearch thus not creating an index on Kibana. Note that I am using fine-grained access control that is why I have a master user and password. In addition, I read some articles that state that I need to include aws-es-proxy but other articles do not even mention it, Please anyone give suggestions which is the best practice. My goal is to simply get my API application Pod logs to Kibana.

Kindly see the logs after using different fluentd images to resolve this issue:

image: quay.io/fluent/fluentd-kubernetes-daemonset

2020-12-16 11:56:04 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2020-12-16 11:56:04 +0000 [info]: starting fluentd-0.12.32
2020-12-16 11:56:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.2'
2020-12-16 11:56:04 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.2'
2020-12-16 11:56:04 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.3'
2020-12-16 11:56:04 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2020-12-16 11:56:04 +0000 [info]: gem 'fluentd' version '0.12.32'
2020-12-16 11:56:04 +0000 [info]: adding match pattern="fluent.**" type="null"
2020-12-16 11:56:04 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2020-12-16 11:57:05 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Invalid Kubernetes API v1 endpoint https://10.100.0.1:443/api: Timed out connecting to server"

image: fluent/fluentd:stable

2020-12-16 12:09:16 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-12-16 12:09:16 +0000 [warn]: [output_docker1] 'time_format' specified without 'time_key', will be ignored
2020-12-16 12:09:16 +0000 [warn]: [output1] 'time_format' specified without 'time_key', will be ignored
2020-12-16 12:09:16 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    @id input1
    @label @mainstream
    port 24224
  </source>
  <filter **>
    @type stdout
  </filter>
  <label @mainstream>
    <match docker.**>
      @type file
      @id output_docker1
      path "/fluentd/log/docker.*.log"
      symlink_path "/fluentd/log/docker.log"
      append true
      time_slice_format %Y%m%d
      time_slice_wait 1m
      time_format %Y%m%dT%H%M%S%z
      <buffer time>
        timekey_wait 1m
        timekey 86400
        path /fluentd/log/docker.*.log
      </buffer>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
    <match **>
      @type file
      @id output1
      path "/fluentd/log/data.*.log"
      symlink_path "/fluentd/log/data.log"
      append true
      time_slice_format %Y%m%d
      time_slice_wait 10m
      time_format %Y%m%dT%H%M%S%z
      <buffer time>
        timekey_wait 10m
        timekey 86400
        path /fluentd/log/data.*.log
      </buffer>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
  </label>
</ROOT>
2020-12-16 12:09:16 +0000 [info]: starting fluentd-1.3.2 pid=6 ruby="2.5.2"
2020-12-16 12:09:16 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2020-12-16 12:09:16 +0000 [info]: gem 'fluentd' version '1.3.2'
2020-12-16 12:09:16 +0000 [info]: adding match in @mainstream pattern="docker.**" type="file"
2020-12-16 12:09:16 +0000 [warn]: #0 [output_docker1] 'time_format' specified without 'time_key', will be ignored
2020-12-16 12:09:16 +0000 [info]: adding match in @mainstream pattern="**" type="file"
2020-12-16 12:09:16 +0000 [warn]: #0 [output1] 'time_format' specified without 'time_key', will be ignored
2020-12-16 12:09:16 +0000 [info]: adding filter pattern="**" type="stdout"
2020-12-16 12:09:16 +0000 [info]: adding source type="forward"
2020-12-16 12:09:16 +0000 [info]: #0 starting fluentd worker pid=16 ppid=6 worker=0
2020-12-16 12:09:16 +0000 [info]: #0 [input1] listening port port=24224 bind="0.0.0.0"
2020-12-16 12:09:16 +0000 [info]: #0 fluentd worker is now running worker=0
2020-12-16 12:09:16.544559051 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
2020-12-16 12:09:16 +0000 [warn]: #0 no patterns matched tag="fluent.info"

fluentd-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      nodeSelector:
        type: persistent
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd:stable
        # quay.io/fluent/fluentd-kubernetes-daemonset
        # fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "https://search-gateway-elk-7xfzdckq2342424ia43242325a.us-east-2.es.amazonaws.com"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          #- name: FLUENT_ELASTICSEARCH_SCHEME
            #value: "http"
          - name: FLUENT_ELASTICSEARCH_USER
            value: "my-username"
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            value: "my-password"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

[cosmo0920]

Duplicated of #244.
AWS Elasticsearch Service requests to use AWS Sign V4 like as other AWS managed services.
This is not issue, just a limitation of AWS Elasticsearch Service.
Closing.

[amin224]

Kindly note that while troubleshooting my issue I did provide the access key in my env but was receiving the same errors. Thank you for the reference I will do more research there to fix my issue.

          #- name: AWS_REGION
            #value: "us-east-2"
          #- name: AWS_ACCESS_KEY_ID
            #value: "..."
          #- name: AWS_SECRET_ACCESS_KEY
            #value: "..."

[cosmo0920]

fluent-plugin-elasticsearch does not handle AWS access key and its secret.
These modified lines are unused.
Instead, you can use dtan4/aws-sign-proxy for proxy request into AWS Elasticsearch Service.
It also can refer as quay.io/dtan4/aws-sign-proxy:latest on Docker repo/release tag.