Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default Logstash index #438

Open
dirnikk opened this issue Nov 10, 2023 · 0 comments
Open

default Logstash index #438

dirnikk opened this issue Nov 10, 2023 · 0 comments

Comments

@dirnikk
Copy link

dirnikk commented Nov 10, 2023

Hi everyone,

Is there a way to edit the name of the default logstash index.
With the config I am able to create desired index called ltb-service and everything is fine.
But there is a default logstash index which also populates to the Opensearch cluster and I am not sure how to change the name of index.
I want to do that because if there are 2 EKS clusters sending logs they ingest on the same index.
Any Idea?
I have the following config file:

replicaCount: 2

serviceAccount:
  name: fluent-bit-o
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/logging-fluent
resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 512Mi
luaScripts:
  utils.lua: |
      -- Function to add an index prefix field
      function add_index_prefix_field(tag, timestamp, record)
        -- Define a list of namespaces to include
        namespaces = {"ltb-service"}

        -- Set the default return code to 0 (no change)
        retcode = 0

        -- Set the prefix value
        prefix = 'eks'

        -- Get the namespace from the record's "kubernetes" field
        namespace = record['kubernetes']['namespace_name']

        -- Check if the namespace is in the list of namespaces to include
        for _, value in ipairs(namespaces) do
          if value == namespace then
            -- Create the index prefix using the prefix value and the namespace name
            index_prefix = prefix .. '_' .. namespace

            -- Add the index prefix field to the record
            record['index_prefix'] = index_prefix

            -- Set the return code to 2 (update record)
            retcode = 2

            -- Break out of the loop
            break
          end
        end

        -- Return the return code, timestamp, and modified record
        return retcode, timestamp, record
      end


config:
  inputs: |
    [INPUT]
        Name tail
        Path  /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
  filters: |
    [FILTER]
        Name                kubernetes
        Match               *
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
    [FILTER]
        Name      lua
        Match     *
        script    /fluent-bit/scripts/utils.lua
        call      add_index_prefix_field
  outputs: |
    [OUTPUT]
        Name            opensearch
        Match           *
        Host            opensearch-ingest.example.com
        Port            443
        TLS             On
        AWS_Auth        On
        Suppress_Type_Name On
        AWS_Region      us-east-1
        Logstash_Format On
        Logstash_Prefix_Key $index_prefix
        Buffer_Size     False
        Time_Key        @timestamp
        Trace_Error     On
        Replace_Dots    On
tolerations:
  - operator: Exists
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant