Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiline logs are not shown as one Splunk log event but each line is shown as a Splunk log event #859

Open
minhpham-westpac opened this issue May 23, 2023 · 0 comments

Comments

@minhpham-westpac
Copy link

What happened:
In the values.yaml for logging, the pod and container are specified as following with test-sys as the namespace, we call it part A:
aks-secrets-store-provider-azure:
from:
pod: test-sys/aks-secrets-store-provider-azure-
container: provider-azure-installer
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
endline: / pod=".+"/
separator: "\n"
flushInterval: 5
sourcetype: kube:secrets-store-provider-azure

The pod aks-secrets-store-provider-azure generates logs as following, we call it part B:

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

  • |
    objectName: xyz-id
    objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

What you expected to happen:
The above multiline log in part B should be displayed in one Splunk log event as the following, we call it part C

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

  • |
    objectName: xyz-id
    objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

But instead Splunk shows each line as a separate log event as the following, we call it part C:

23/05/2023 17:00:00.123 C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
23/05/2023 17:00:00.124 array:
23/05/2023 17:00:00.125 - |
23/05/2023 17:00:00.126 objectName: xyz-key
....
23/05/2023 17:00:00.200 > pod="kube-system/aks-cluster-compliance-rcg87"

How to reproduce it (as minimally and precisely as possible):
Using the above config for the AKS container in part A, and the input is the multiline log in part B,

Anything else we need to know?:
I follow exactly the instructions in the Splunk connect for Kubernetes https://github.com/splunk/splunk-connect-for-kubernetes/issues?q=is%3Aissue+is%3Aclosed+multiline, and the multiline log is not shown as one Splunk log event. I do not know why, please help me to have the function working. Thanks very much for your help in advance.

Environment:

  • Kubernetes version (use kubectl version): Kubernetes v1.26.3
  • Ruby version (use ruby --version): Not use Ruby at all
  • OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server, VERSION="7.9 (Maipo)"
  • Splunk version:
  • Splunk Connect for Kubernetes helm chart version: Splunk connect for k8s 1.5.3
  • Others:

Please get back to me if you have further questions, or clarification. I am looking forward to your solutions and Thanks very much for your help in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant