bug: Configuration not being live-reloaded

[bruce-y]

Describe the issue

I'm not sure if this is the right repository to open this issue, but the problem seems to be in the live-reloading feature in the kubesphere/fluent-bit image. It seems the configuration is not updated until I forcibly restart the pod.

To Reproduce

We updated the host value in the cluster output:

Before:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  labels:
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: my-output
spec:
  alias: my_pattern
  customPlugin:
    config: |
      Name            http
      Match           my_pattern
      Host            ingress.coralogix.com
      Port            443
      URI             /logs/v1/singles
      Format          json_lines
      TLS             On
      Header          Authorization Bearer <redacted>
      compress        gzip
      Retry_Limit     10
  match: my_pattern

After:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  labels:
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: myoutput
spec:
  alias: mypattern
  customPlugin:
    config: |
      Name            http
      Match           mypattern
      Host            ingress.cx498-aws-us-west-2.coralogix.com
      Port            443
      URI             /logs/v1/singles
      Format          json_lines
      TLS             On
      Header          Authorization Bearer <redacted>
      compress        gzip
      Retry_Limit     10
  match: mypattern

However, we were still observing that in the fluent-bit logs, it was trying to send the logs to:

fluent-bit-45lzx fluent-bit [2024/02/05 23:15:00] [error] [output:http:mypattern] ingress.coralogix.com:443, HTTP status=403

The config generated by the fluent-operator was correct as well:

[Output]
    Match    mypattern
    Alias    mypattern
    Name            http
    Match           mypattern
    Host            ingress.cx498-aws-us-west-2.coralogix.com
    Port            443
    URI             /logs/v1/singles
    Format          json_lines
    TLS             On
    Header          Authorization Bearer <redacted>
    compress        gzip
    Retry_Limit     10

This only resolved after I did a kubectl rollout restart daemonset fluent-bit and then it began sending via the correct configuration:

fluent-bit-45lzx fluent-bit [2024/02/06 00:00:52] [ info] [output:http:mypattern] ingress.cx498-aws-us-west-2.coralogix.com:443, HTTP status=200

Expected behavior

The new configuration should be loaded by the fluent-bit pods without a forced restart.

Your Environment

- Fluent Operator version: docker.io/kubesphere/fluent-operator:v2.7.0
- Fluent Bit version: docker.io/kubesphere/fluent-bit:v2.2.2
- Container Runtime: containerd
- Operating system: Amazon Linux 2023
- Kernel version: 5.10.205-195.804.amzn2.x86_64

How did you install fluent operator?

I'm using the helm chart. This is deployed to AWS EKS.

Additional context

No response

[tomsiewert]

I see something similar with changes in a Lua script / ClusterFilter, but not even a DaemonSet restart helps there. The rendered fluentbit.conf and other ConfigMaps / Secrets look fine. I have to re-create the whole Fluentbit resource to get fluent-bit picking up the changes.