new pod logstream not getting created with mounted-file option #289

vkadi · 2021-11-15T17:20:19Z

I am seeing the issue with CW logstreams not getting created when using the "mounted-file" to ingest the logs from file in the container.
I picked the sample config from examples folder and able to reproduce it , below is the demo config what I have used.

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: default
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: ubuntu
name: test-container
command:
- bash
- -c
- while true; do echo date -R [INFO] "Random welcome number $((var++)) to file"; sleep 2; [[ $(($var % 100)) == 0 ]] && :> /var/log/welcome.log ;done > /var/log/welcome.log
volumeMounts:
- mountPath: /var/log
name: logs
- mountPath: /host-root
name: root
volumes:
- name: logs
emptyDir: {}
- name: root
hostPath:
path: /

Sometime when we refresh the fluent then we will see the logstreams but its not consistent either. Anyone using the "mounted-file" option heavily and saw this issue. Any guidance will be a great help around this area.
Note : Using the latest KFO version (1.16.1)

The text was updated successfully, but these errors were encountered:

vkadi · 2021-11-16T20:25:34Z

@Cryptophobia Can you please help with the issue , seems like the new container information is not updated in "fluent.conf" until the fluent is refreshed manually.

Cryptophobia · 2021-11-17T16:35:34Z

@vkadi , can you share the configMap or CRD fluentdconfig that you are using for the above testing container?

Are you using type @mounted-file? Are you following the README here: https://github.com/vmware/kube-fluentd-operator#i-have-a-legacy-container-that-logs-to-varloghttpdaccesslog

Also it is possible, when your configuration changes the fluentd container needs to be restarted as gracefulReload may not be enough. Let me know if you can provide the detail above and I can test with a similar configmap or CRD.

vkadi · 2021-11-17T20:02:41Z

@Cryptophobia Yes, I am using the "mounted-file" and this is how my configmap looks for the testing pod -

<source>
   @type mounted-file
   path /var/log/welcome.log
   labels msg= nginx, _container=test-container
</source>
<match **>
   @type cloudwatch_logs
   auto_create_stream true
   log_stream_name_key stream_name
   remove_log_stream_name_key true
   retention_in_days 90
   region "#{ENV['AWS_REGION']}"
   <buffer>
     @type file
     path /var/log/fluentd
     flush_interval 5s
     flush_thread_count 8
     chunk_limit_size 2m
     queued_chunks_limit_size 32
     retry_forever true
   </buffer>
   log_group_name /aws/eks/<cluster_name>/cluster/namespace/default
</match>

sabdalla80 · 2021-11-17T20:59:32Z

@Cryptophobia
I am having a similar problem with mounted-file as @vkadi. I believe the problem isn't in configMap changes, this part works as expected, reloader kicks in and invokes controller::RunOnce() loop and re-generates the proper ConfigMaps in addition to fluent.conf.
However, the problem seems to be when the pod in a deployment is replaced/restarted, the reloader seems to be unaware and thus it does not re-generate or update the proper fluent.conf and configmap files again and hence it never executes the mountedfile::Process() and also it won't create the cloudwatch logstream with the new pod name as a result.

Note that fluentd logs does show the new pod such as:
2021-11-17 16:51:15 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/nginx-deployment-f697bcfb-8b2jm_default_test-container-a13490af26e0e63a6eeafa884c66b6e7891835fc204fa2c259f5e3105a946496.log 2021-11-17 16:51:40 +0000 [info]: #0 detected rotation of /var/lib/kubelet/pods/601ac67a-0639-4f28-979b-cf21151257bd/volumes/kubernetes.io~empty-dir/logs/welcome.log; waiting 5 seconds 2021-11-17 16:51:41 +0000 [info]: #0 [in_tail_container_logs] detected rotation of /var/log/containers/nginx-deployment-f697bcfb-clqgw_default_test-container-664c91264cbc2b835b9a2ab1260b9c1c6d398f1ff419907cc5b243273b5efa97.log; waiting 5 seconds

Cryptophobia · 2021-11-17T22:38:33Z

Thank you @sabdalla80 , that description helps a lot! I think in this case maybe we are not doing it correctly. Since the config file has not change, we only force fluentd reload when config hashes change. We need to force gracefulReload on fluentd even when configmap does not change but the mountedFile pod is restarted/replaced.

This sounds like the issue here. This also explains why KFO sometimes does not work with job type pods.

Cryptophobia · 2021-11-17T22:41:50Z

I think this is related to the issues in #253

sabdalla80 · 2021-11-18T13:42:13Z

Thank you @sabdalla80 , that description helps a lot! I think in this case maybe we are not doing it correctly. Since the config file has not change, we only force fluentd reload when config hashes change. We need to force gracefulReload on fluentd even when configmap does not change but the mountedFile pod is restarted/replaced.

This sounds like the issue here. This also explains why KFO sometimes does not work with job type pods.

@Cryptophobia Thank you for the response and feedback. It sounds like others having same issue, I know this issue is currently impacting me and I can use a quick fix. Do you know if you have an ETA for the fix? I would be glad to assist as I already went down this path of learning the insides of the code and got hung up on how to detect pod changes (I am not 100% sure, but it sounds like it's done in KubeInformer). Please let me know as I can help with testing the fix as well since I already have an environment with this issue ready for testing.

Cryptophobia · 2021-11-18T17:32:39Z

@sabdalla80 , yes I think we need to find some place in kubeInformer.go (config-reloader/datasource/kube_informer.go) where we can do this check via the sharedInformers for the pods. If pods are getting restarted in a namespaces, config-reloader needs to be smart enough to issue new reload to fluentd. This is where we can get a list of pods from namespace config struct that we build.

@sabdalla80 Before we start looking at code fix can you verify if this is not a regression for me? There are a couple lines of code were deleted that were thought to be useless. Can you run the v1.15.0 docker image and verify you are seeing the same problems?

The line deleted in question is this commit: https://github.com/vmware/kube-fluentd-operator/pull/227/files

sabdalla80 · 2021-11-18T17:39:28Z

@sabdalla80 , yes I think we need to find some place in kubeInformer.go (config-reloader/datasource/kube_informer.go) where we can do this check via the sharedInformers for the pods. If pods are getting restarted in a namespaces, config-reloader needs to be smart enough to issue new reload to fluentd. This is where we can get a list of pods from namespace config struct that we build.

@sabdalla80 Before we start looking at code fix can you verify if this is not a regression for me? There are a couple lines of code were deleted that were thought to be useless. Can you run the v1.15.0 docker image and verify you are seeing the same problems?

The line deleted in question is this commit: https://github.com/vmware/kube-fluentd-operator/pull/227/files

@Cryptophobia I tried it against 1.15.0 and 1.13.0, it has the same issue.

Cryptophobia · 2021-11-18T17:47:43Z

I just looked at the code and I have an idea. We can use an hasher to get a unique has for the nsConfig struct each time RunOnce() loop runs. This will avoid us having to reload each time new pods restart or using kubeInformer at all. We just build namespace representation using our current way.

If the podlist slice representation in nsConfig.MiniContainers changes then the hash value of the whole nsConfig struct will change and we can issue reload.

We can use this for our hasher:
https://github.com/mitchellh/hashstructure

EDIT: The true problem stems from the fact that mount-file names change based on the container names, but forcing more reloads will make KFO more resilient and this is the best way to solve the problem I think.

Cryptophobia · 2021-11-18T20:08:05Z

@sabdalla80, thank you for testing. After I do some local testing, I will build a new test image today.

sabdalla80 · 2021-11-18T20:23:03Z

@sabdalla80, thank you for testing. After I do some local testing, I will build a new test image today.

This would be spectacular. @Cryptophobia If you can share any code/branch you have started, I can contribute to it if it helps to bring to completion.

Cryptophobia · 2021-11-19T21:38:02Z

@sabdalla80 , When I do a simple test with 3 pods in a deployment. I cannot reproduce this. You are saying the mountedfile:Process() is not being called. Can you be more specific and in which part of the code?

sabdalla80 · 2021-11-19T21:41:49Z

@Cryptophobia Are you able to see main control loop re-run when you restart/replace a pod in the deployment? Does it re-generate fluentd.conf file?

Cryptophobia · 2021-11-19T21:44:32Z

No, the main control loop does not rerun. But I see fluentd picks up new pods and new container names when the containers restarts/scale-up or whatever. This seems like a race-condition with logstream for CW?

Cryptophobia · 2021-11-19T21:47:44Z

@sabdalla80 , do you think if we force fluentd to reload this will be fixed? This seems like CW logstreams expect to find new names of pods and it is not being forced because of race condition?

sabdalla80 · 2021-11-19T21:51:33Z

@Cryptophobia In intial deployment things work fine, the control loop generates fluentd.conf and proper CM files for a service. On pod restart, even though fluentd indeed sees the new pod, the fluend.conf and CM files are not re-generated, they become stale and don't point correctly to the new pods.
To answer your question, from the behavior I have seen thus far, I do believe forcing to reload will fix the problem. As a matter of fact, I tested this theory by restarting fluentd pod manually and it seems to update things correctly. Is there a way we can code this and test it? I would have tried it if I have good understanding of the code that is initiating the detection of pod changes.

For example: this is what's generated in fluent.conf file in deployment for mounted-file pod. When mounted-file pod restarts, this file isn't changed.

#################
# Namespace pre-processing
#################
<source>
  @type tail
  path /var/lib/kubelet/pods/05fa8d22-a910-4edc-8004-6a1ca18a252e/volumes/kubernetes.io~empty-dir/logs/welcome.log
  pos_file /var/log/kfotail-23b81aaedcbe0520c93c4396c956c89b8136408d.pos
  read_from_head true
  tag kube.default.nginx-deployment-f697bcfb-ct5k5.test-container-23b81aaedcbe0520c93c4396c956c89b8136408d

  <parse>
    @type none
  </parse>
</source>

<filter kube.default.nginx-deployment-f697bcfb-ct5k5.test-container-23b81aaedcbe0520c93c4396c956c89b8136408d>
  @type record_modifier
  remove_keys dummy_

  <record>
    dummy_ ${record['stream']='/var/log/welcome.log'; record['kubernetes']={'container_image'=>'ubuntu','container_name'=>'test-container','host'=>'ip-10-0-29-130.ec2.internal','namespace_name'=>'default','pod_id'=>'05fa8d22-a910-4edc-8004-6a1ca18a252e','pod_name'=>'nginx-deployment-f697bcfb-ct5k5'}; record['docker']={'container_id'=>'docker://fd034ffb705c6be7a6d726139622955e4d999b2d532a3497d7c97233c62ccbea'}; record['container_info']='3039a456dcf33176ead2f0e1223137d0ac6a2e3e'; record['kubernetes']['labels']={'app'=>'nginx','pod-template-hash'=>'f697bcfb'}; record['kubernetes']['namespace_labels']={}}
  </record>
</filter>

Cryptophobia · 2021-11-19T22:06:31Z

@sabdalla80 , thank you for this. I think I understand the problem. The CW plugin requires that the container name be known ahead of time so even though we make correct fluentd configs from reloader when fluentd does not restart then the CW plugin configuration is not the correct one loaded. I am building a container for you to test this theory. The fix is basically to reload anytime we have new pods. I think this idea will make KFO more robust anyways. We will see in practice if this is desired.

sabdalla80 · 2021-11-19T22:15:10Z

@sabdalla80 , thank you for this. I think I understand the problem. The CW plugin requires that the container name be known ahead of time so even though we make correct fluentd configs from reloader when fluentd does not restart then the CW plugin configuration is not the correct one loaded. I am building a container for you to test this theory. The fix is basically to reload anytime we have new pods. I think this idea will make KFO more robust anyways. We will see in practice if this is desired.

It would be awesome. You are spot on in your description.

Cryptophobia · 2021-11-19T22:40:10Z

@sabdalla80 , I still can't produce a valid test with the similar preprocessing block as your config. Can you post the fluentd-config you are using for the namespace and the deployment yaml that you are deploying for this mount-file test?

Cryptophobia · 2021-11-19T22:40:47Z

Can you try testing with this image when you have some time?

vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-1

EDIT: Image is still being pushed up, give it 2 mins.

Cryptophobia · 2021-11-19T22:56:41Z

@sabdalla80 , I am able to reproduce the preprocessing block now. I see when I scale up from 3 to 6 pods, the preprocessing still stay to 3 blocks...

sabdalla80 · 2021-11-19T23:01:54Z

@Cryptophobia

with vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-1

I did see reloader log this event this time around on mounted-file pod restart. But, I did not see fluentd.conf file update. Are you not running the whole chain of events such as generator::renderToDisk in your new changes?

time="2021-11-19T22:49:05Z" level=debug msg="Config hashes returned in RunOnce loop: map[abdas1-pega-58:8c3d1db1c72be76ae2299c726880b349ee7ee82d calico-system:7b0343ff7ee57ab27aef853cbfab80c49bcc9643 default:468b29a540388037bedc0ba2c9324d8e0240695b ext-secrets:8fc4450b5f01d9eb4a007f82aa1013df9a56aa1a gloo-system:155cb45bff838838d4514b6680b27a1aae5bec32 istio-system:d345dfc9f351f907c562df9fd0a537d27605b57b k8s-infra-components:6b89ba488815961fc3446e61848d09d5dc5b9f1c k8s-infra-operator:95862c8f02da78a629e30e44a1e8f4093b00a6bd kube-public:c8e51186853ef96e15db4c46ebbe0b6b5e0c4a3f kube-system:8ecffb9f57c8a0c48eeadbc36c9f247d669fc5d2 logging:71cf57b46a837f2b5f0bd616cdf12b2f245d2f0a metacontroller:0d91bcef8e1189aa3d10d0656d471268c039ec03 srs:1bedbb48a6f3eddab391c51c906e6355b4307656 tigera-operator:30b3bcb04653fc02d66c5579a320448010c942c2]"

Cryptophobia · 2021-11-20T00:16:40Z

I think I solved it @sabdalla80 !

By making a function with a channel in the Run() loop (moving it out of RunOnce()) so that it does the check now and does not rely on the KubeInformer update channels, but does it outside of the informers. This was to save replicating the code across all of the different type of informers. It's a hacky solution, but if it works, it will solve all the preprocessing race conditions problems we may have in future too. 😄

Please test when you can:

vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-2

Here was the solution:

	// checkIfNeedsReload will verify if new containers neeed to preprocess
	checkIfNeedsRunOnce := func() (needsRun <-chan bool) {
		needsForceRunOnce := make(chan bool, 1)
		allConfigNamespaces, _ := c.Datasource.GetNamespaces(ctx)
		for _, nsConfig := range allConfigNamespaces {
			// if there are new/restarted pods in the configed namespace, we should reload just in case:
			// mounted-file fix, mounted file path changes based on container names:
			// this may fix race condition in logstream plugins that depend on container names?
			if !util.AreStructureHashEqual(nsConfig.MiniContainers, nsConfig.MiniContainers) {
				needsForceRunOnce <- true
			} else {
				needsForceRunOnce <- false
			}
		}
		return needsForceRunOnce
	}
	```

Cryptophobia · 2021-11-20T00:20:55Z

@sabdalla80 I will wait for your test with container before I make PR:

vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-2

sabdalla80 · 2021-11-20T00:35:04Z

@sabdalla80 I will wait for your test with container before I make PR:
vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-2

@Cryptophobia I am still seeing the same behavior. It seems as if /fluentd/etc/fluent.conf and the .conf files for other namespaces remain stale when mount-file pods are replaced.
Is it possible kubeInformer isn't detecting the replacement change of pods to begin with? The reason I say this is I would have expected to see controller and Generator events to show up in reloader logs when I bounce a pod. Furthermore, I have tried adding log messages in the kubeInformer functions such as discoverNamespaces() and GetNameSpaces() and I did not see them run when mounted_file pod is replaced.

In contrast to configMap::detectConfigMapName() which seems to be listening all the time, the kubeInformer::GetNamespaces() does not seem to be listening. It appears as if KubeInformer needs some listening logic similar to Configmap.go. The latter seems to have updateCh logic and the former doesn't. Could that be the issue?

sabdalla80 · 2021-11-24T04:42:09Z

@Cryptophobia Great news! I have verified that sabdalla80-test-3 image worked successfully for adding, updating, and deleting pods. Do you plan on merging to master today? I am looking forward to the new image. Thanks again for your help!

Cryptophobia · 2021-11-24T05:36:23Z

@sabdalla80, Great news! Tomorrow, I will make and publish a new minor release v1.16.2

Cryptophobia · 2021-11-24T18:01:24Z

@sabdalla80 , apologies to bother you today. I had a realization this morning that we do not need to compare hashes of all pods and store those at all. We can just rely on config hashes changing per namespace, since preProcessors are part of renderToDisk function the config for each namespace will include the preprocessing directives.

I removed a bunch of useless code I had and now it is a much leaner solution (no more hashing structures of pods that may not be related to any configs).

Can you please just verify that it all works when you scale up and down with configmaps type? I have verified crd type works again. Would appreciate a second pair of eyes on this before making the PR and release.

vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-4

sabdalla80 · 2021-11-24T19:01:33Z

Certainly! I Will do shortly and will report back.

…

________________________________ From: AMO ❤️☕ ***@***.***> Sent: Wednesday, November 24, 2021 1:01:34 PM To: vmware/kube-fluentd-operator ***@***.***> Cc: Sam A. ***@***.***>; Mention ***@***.***> Subject: Re: [vmware/kube-fluentd-operator] new pod logstream not getting created with mounted-file option (Issue #289) @sabdalla80<https://github.com/sabdalla80> , apologies to bother you today. I had a realization this morning that we do not need to compare hashes of all pods and store those at all. We can just rely on config hashes changing per namespace, since preProcessors are part of renderToDisk function the config for each namespace will include the preprocessing directives. I removed a bunch of useless code I had and now it is a much leaner solution (no more hashing structures of pods that may not be related to any configs). Can you please just verify that it all works when you scale up and down with configmaps type? I have verified crd type works again. Would appreciate a second pair of eyes on this before making the PR and release. vmwaresaas.jfrog.io/vdp-public/upstream/kube-fluentd-operator:sabdalla80-test-4 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#289 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAUNLXHHKQDDYDV7NFGJTXTUNUR75ANCNFSM5ICHBFKA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

- add kube pod informer for pods add/rm in namespace we monitor, rerun preprocess configs - fixes vmware#289 - fixes vmware#253 - fixes issue with respecting `c.cfg.AnnotConfigmapName` on configmap namespaces discovery - introduces `hashstructure` package for smarter hashing of changing objects - make smarter hashing for `allConfigsHash` metahashing (without `reflect.DeepEqual()`) Signed-off-by: Anton Ouzounov <aouzounov@vmware.com>

sabdalla80 · 2021-11-24T20:16:17Z

@Cryptophobia I like the cleanup - looks good.
I have also verified sabdalla80-test-4 works correctly with scaling up and down of mounted_file pods as well as replacing/updating the pods.

Cryptophobia · 2021-11-24T23:06:06Z

New v1.16.2 release is out. New image is now docker.io/vmware/kube-fluentd-operator:v1.16.2

Cryptophobia · 2021-12-03T23:17:43Z

Hey @sabdalla80,

Unfortunately, this needs to be reverted for now. We noticed that too many gracefulReload calls are causing the fluentd plugins to break and fluentd gets stuck. Fluentd sits in a bad state so in very huge clusters with lots of configs this is a problem.

I think fluentd just gets in a bad state if too many gracefulReloads come in all at once. I have no other explanation. Which means we need to be way smarter when we decide to reload. We need to be even more deliberate… We can’t just reload all the time when new pods come in and out… we will need to somehow keep track if those pods are in the preprocessing directives, if they are, then reload… I will most likely need to modify preprocessor function to return actual pod names and container ids of the pods that are in the preprocessing config. That way the controller will track them. Then when pods come in only reload if that pod matches the previous config's container ids.

I will go back to the drawing board and look at other ways of fixing this.

sabdalla80 · 2021-12-06T13:57:20Z

@Cryptophobia Sorry about the late response, this got lost in my email box somehow. I would like to collaborate with you on the fix as needed. Getting this right is critical to us. Besides the increase in reloads, how did you find out that fluentd was getting hung? Ideally, on the update front, we don't want to reload everytime a pod changes, we just want to reload only if a mounted_file pod has been updated.

Cryptophobia · 2021-12-06T18:50:02Z

Besides the increase in reloads, how did you find out that fluentd was getting hung?

Log volumes got really low (see picture above) and when inspecting the fluentd containers, we saw that we don't get normal debug/info logs from fluentd but rather it was stuck after the gracefulReload API call. Upon further inspection of the logs in the fluentd container, it looks like fluentd is taking a lot of processing power to release the buffers/load/unload plugins and then becomes stuck in that data pipeline flush code (suspecting this is the case). Fluentd cannot keep up with all of the API calls to reload when using too many plugins and too many configs. It could also be that it tries to flush too many file buffers from our fluentd configuration files per namespaces... (another possibility).

Ideally, on the update front, we don't want to reload every time a pod changes, we just want to reload only if a mounted_file pod has been updated.

Yes, this is exactly the point. We need a better solution that only reloads when mount_file pods are updated. Only call the update event in the pod reloader when a container that has mount_file directive (in preprocessors) to regenerate configs. This will lower the number of reload calls to the absolute requirement to make it all work without DOS on fluentd pod's API.

sabdalla80 · 2022-01-05T15:28:52Z

@Cryptophobia I am going to start working on it today. I am going to figure out how to reload only when mounted_file pod restarted. If you already thought about how the code may look like, please share. I will also need your help when fix is in so you can run the same tests you ran to ensure fluentd is not getting hung with too many reload calls.

sabdalla80 · 2022-01-07T21:27:28Z

@Cryptophobia Can you provide more input on the behavior you saw, I was not able to reproduce. I deployed a mounted-file pod and a standard pod (not mounted-file), and I observed the standard pod does not actually execute graceful reload calls (but it does go thru the RunOnce() controller function). The mounted-file pod does execute graceful reloads on pod changes per the fix.
Related to extra consumption/processing and for the standard(non-mounted) pods, I am wondering if it's due to the unnecessary processing of the renderToDisk() function within RunOnce() loop? Ideally for non-mounted pods, we don't want to run controller loop at all. Do you know if there is a way we can flag for that right in the kubeInformer module at the time when we detect pod changes? In other words, in KuberInformer, are we able to find the namespace config associated with the pod that just changed and based on the mounted-file type, we can run updateChan?

On a side note: On ConfigMap changes or Pod changes(now), RunOnce() control loop seems to scan all namespaces/configs everytime(I think this should be an enhancement where it only runs against the namespace that changed)

sabdalla80 · 2022-01-12T00:29:55Z

@Cryptophobia I made some changes so that we only run controller and reload when pod changes is associated with namespace/config that has mounted-file type. I am doing this in kube_informer and not in controller to avoid the RenderToDisk() call. The PR branch is here: #306 (Thank you @kadiv1). Could you please test the changes by running your metrics tests from last time to see if log volumes are ok and there are no extra graceful reloads? If this works and does the trick, we can think of more elegant solution if need be.

I appreciate your help on this to help unblock me as I am not able to proceed forward until I verify there are no issues with v16.2

sabdalla80 · 2022-02-02T21:16:46Z

@Cryptophobia Can you help reviewing the above changes?

javiercri · 2022-05-30T11:14:13Z

Current solution in 1.16.5 is not affordable for big clusters.
Issue revert in 1.16.6, we will need to think in a better approach.

shrxx · 2022-08-03T16:19:02Z

Hi All
So, there is no solution so far? I have tried I think every offered solution in this thread but still no luck.
@javiercri maybe there is a workaround? Could you suggest something?

) Signed-off-by: huskykurt <rkmwiaim1@gmail.com>

Signed-off-by: huskykurt <rkmwiaim1@gmail.com>

…d-file is created or deleted (#347) * Reload configuration only if changed pod with mounted labels (#289) * add test for util/Match function (#289) Signed-off-by: huskykurt <rkmwiaim1@gmail.com>

Cryptophobia self-assigned this Nov 17, 2021

Cryptophobia added bug Something isn't working good first issue Good for newcomers critical critical bug should be fixed (priority) labels Nov 17, 2021

Cryptophobia mentioned this issue Nov 24, 2021

fix: react to pod events to regen preprocess conf #293

Merged

Cryptophobia closed this as completed in #293 Nov 24, 2021

Cryptophobia reopened this Dec 3, 2021

Cryptophobia mentioned this issue Dec 8, 2021

Fluentd stops processing the logs if a pod is restarted #296

Closed

Cryptophobia mentioned this issue Feb 3, 2022

Issue#289 Mounted file issue fix part-2 #306

Merged

javiercri closed this as completed Mar 1, 2022

javiercri mentioned this issue May 30, 2022

Revert issue big clusters #325

Merged

javiercri reopened this May 30, 2022

This was referenced Oct 19, 2022

Reload Configuration only when a pod matching the label set in mounted-file is created or deleted #345

Closed

Reload Configuration only when a pod matching the label set in mounted-file is created or deleted. #346

Closed

huskykurt added a commit to huskykurt/kube-fluentd-operator that referenced this issue Oct 20, 2022

Reload configuration only if changed pod with mounted labels (vmware#289

48cfbed

) Signed-off-by: huskykurt <rkmwiaim1@gmail.com>

huskykurt mentioned this issue Oct 20, 2022

Reload Configuration only when a pod matching the label set in mounted-file is created or deleted #347

Merged

huskykurt added a commit to huskykurt/kube-fluentd-operator that referenced this issue Dec 14, 2022

add test for util/Match function (vmware#289)

124278e

Signed-off-by: huskykurt <rkmwiaim1@gmail.com>

new pod logstream not getting created with mounted-file option #289

new pod logstream not getting created with mounted-file option #289

Comments

vkadi commented Nov 15, 2021 • edited

I am seeing the issue with CW logstreams not getting created when using the "mounted-file" to ingest the logs from file in the container. I picked the sample config from examples folder and able to reproduce it , below is the demo config what I have used.

vkadi commented Nov 16, 2021

Cryptophobia commented Nov 17, 2021

vkadi commented Nov 17, 2021 • edited

sabdalla80 commented Nov 17, 2021 • edited

Cryptophobia commented Nov 17, 2021

Cryptophobia commented Nov 17, 2021

sabdalla80 commented Nov 18, 2021

Cryptophobia commented Nov 18, 2021

sabdalla80 commented Nov 18, 2021 • edited

Cryptophobia commented Nov 18, 2021 • edited

Cryptophobia commented Nov 18, 2021

sabdalla80 commented Nov 18, 2021 • edited

Cryptophobia commented Nov 19, 2021

sabdalla80 commented Nov 19, 2021 • edited

Cryptophobia commented Nov 19, 2021

Cryptophobia commented Nov 19, 2021

sabdalla80 commented Nov 19, 2021 • edited

Cryptophobia commented Nov 19, 2021

sabdalla80 commented Nov 19, 2021

Cryptophobia commented Nov 19, 2021

Cryptophobia commented Nov 19, 2021 • edited

Cryptophobia commented Nov 19, 2021

sabdalla80 commented Nov 19, 2021 • edited

Cryptophobia commented Nov 20, 2021

Cryptophobia commented Nov 20, 2021

sabdalla80 commented Nov 20, 2021 • edited

sabdalla80 commented Nov 24, 2021 • edited

Cryptophobia commented Nov 24, 2021

Cryptophobia commented Nov 24, 2021

sabdalla80 commented Nov 24, 2021 via email • edited

sabdalla80 commented Nov 24, 2021

Cryptophobia commented Nov 24, 2021

Cryptophobia commented Dec 3, 2021

sabdalla80 commented Dec 6, 2021 • edited

Cryptophobia commented Dec 6, 2021 • edited

sabdalla80 commented Jan 5, 2022 • edited

sabdalla80 commented Jan 7, 2022 • edited

sabdalla80 commented Jan 12, 2022 • edited

sabdalla80 commented Feb 2, 2022

javiercri commented May 30, 2022

shrxx commented Aug 3, 2022

vkadi commented Nov 15, 2021 •

edited

I am seeing the issue with CW logstreams not getting created when using the "mounted-file" to ingest the logs from file in the container.
I picked the sample config from examples folder and able to reproduce it , below is the demo config what I have used.

vkadi commented Nov 17, 2021 •

edited

sabdalla80 commented Nov 17, 2021 •

edited

sabdalla80 commented Nov 18, 2021 •

edited

Cryptophobia commented Nov 18, 2021 •

edited

sabdalla80 commented Nov 18, 2021 •

edited

sabdalla80 commented Nov 19, 2021 •

edited

sabdalla80 commented Nov 19, 2021 •

edited

Cryptophobia commented Nov 19, 2021 •

edited

sabdalla80 commented Nov 19, 2021 •

edited

sabdalla80 commented Nov 20, 2021 •

edited

sabdalla80 commented Nov 24, 2021 •

edited

sabdalla80 commented Nov 24, 2021 via email •

edited

sabdalla80 commented Dec 6, 2021 •

edited

Cryptophobia commented Dec 6, 2021 •

edited

sabdalla80 commented Jan 5, 2022 •

edited

sabdalla80 commented Jan 7, 2022 •

edited

sabdalla80 commented Jan 12, 2022 •

edited