You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is there a way to edit the name of the default logstash index.
With the config I am able to create desired index called ltb-service and everything is fine.
But there is a default logstash index which also populates to the Opensearch cluster and I am not sure how to change the name of index.
I want to do that because if there are 2 EKS clusters sending logs they ingest on the same index.
Any Idea?
I have the following config file:
replicaCount: 2
serviceAccount:
name: fluent-bit-o
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/logging-fluent
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 200m
memory: 512Mi
luaScripts:
utils.lua: |
-- Function to add an index prefix field
function add_index_prefix_field(tag, timestamp, record)
-- Define a list of namespaces to include
namespaces = {"ltb-service"}
-- Set the default return code to 0 (no change)
retcode = 0
-- Set the prefix value
prefix = 'eks'
-- Get the namespace from the record's "kubernetes" field
namespace = record['kubernetes']['namespace_name']
-- Check if the namespace is in the list of namespaces to include
for _, value in ipairs(namespaces) do
if value == namespace then
-- Create the index prefix using the prefix value and the namespace name
index_prefix = prefix .. '_' .. namespace
-- Add the index prefix field to the record
record['index_prefix'] = index_prefix
-- Set the return code to 2 (update record)
retcode = 2
-- Break out of the loop
break
end
end
-- Return the return code, timestamp, and modified record
return retcode, timestamp, record
end
config:
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
filters: |
[FILTER]
Name kubernetes
Match *
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name lua
Match *
script /fluent-bit/scripts/utils.lua
call add_index_prefix_field
outputs: |
[OUTPUT]
Name opensearch
Match *
Host opensearch-ingest.example.com
Port 443
TLS On
AWS_Auth On
Suppress_Type_Name On
AWS_Region us-east-1
Logstash_Format On
Logstash_Prefix_Key $index_prefix
Buffer_Size False
Time_Key @timestamp
Trace_Error On
Replace_Dots On
tolerations:
- operator: Exists
The text was updated successfully, but these errors were encountered:
Hi everyone,
Is there a way to edit the name of the default
logstash
index.With the config I am able to create desired index called
ltb-service
and everything is fine.But there is a default
logstash
index which also populates to the Opensearch cluster and I am not sure how to change the name of index.I want to do that because if there are 2 EKS clusters sending logs they ingest on the same index.
Any Idea?
I have the following config file:
The text was updated successfully, but these errors were encountered: