Ways to deal with very large configs

[markusthoemmes]

Is your feature request related to a problem? Please describe.

Kubernetes Secrets are limited in size by what's configured in etcd. Frequently, that limit is 1MB. Currently, the entire configuration for fluentbit is eventually written to a single secret, even if namespaced CRDs are used. That secret is then mounted into the fluentbit containers and eventually read by the fluentbit process to act on it.

Due to the intermediate write to a secret, very large configs eventually fail to be processed because the secret can no longer be written to.

Describe the solution you'd like

There's essentially 2 ways I can think about fixing this, one being a "not really a fix" 😅 .

1. The simple bandaid: The fluentbit configuration being cleartext with a lot of repetitive words makes it very compressable. In one cluster, we have a config that's 501K in size. Compressing it via gzip drops the size to 34K, just 6,7% of the initial size! Obviously, this is not a "solution" to the initial problem but rather a way to kick the stone down the street for longer. Eventually, at enough scale, the problem will reappear. It is however fairly easy to implement in that the controller would only have to learn how to write the secret in a gzipped way and the watcher would have to learn how to deal with reading gzipped
2. Moving final config generation to the watcher: In order to "properly" fix the issue, we could move the config generation into the watchers themselves. They've have to be able to read all relevant CRDs from the K8s API and essentially, the code that's currently writing the secret would move into the watcher to generate the config locally to each fluentbit container. As a consequence, RBAC for the fluentbit containers would get wider.

I'd love to get feedback on either of these small outlines. If acceptable, I'd be happy to work either approach into a full-blown proposal and contribute the respective necessary changes.

Additional context

No response

[benjaminhuo]

The second option looks good, Prometheus operator uses another approach as I know
@wanjunlei @wenchajun what do you think?

[wenchajun]

I also think the second one is better.

[benjaminhuo]

As mentioned by @wanjunlei , by adding the config generation logic into the watcher sidecar, we can also try to enable the config hot reload feature of fluentbit and remove the logic in the watcher to restart the fluentbit process whenever config changes. @markusthoemmes

@wanjunlei do you have any tips for @markusthoemmes to implement this?

[markusthoemmes]

Config generation itself should be fairly straightforward for me to implement in the watcher. I'd imagine that's a bit of RBAC + moving the code + adding a few flags to control new/old behavior.

Hot reload would indeed be an interesting followup, but I think we should keep those concerns separate.

[markusthoemmes]

This made me wonder actually: Once config generation is inside the watcher, do we need the fluentbit-manager itself at that point? We could pass FluentBitConfigName and NamespacedFluentBitCfgSelector to the watcher via flags and then "just" apply a DaemonSet with the fluentbit config we desire (in "native" K8s) vs. mirroring all of the fields in the ClusterFluentBitConfig 🤔 .

I guess it's somewhat orthogonal but I'd be interested in exploring that angle potentially as well. It would simplify the operator and give the user more power for deciding how to deploy fluent bit under the hood.

[jeff303]

Another idea would be to support the YAML configuration syntax for Fluent Bit, which itself supports includes, so that large config sections can potentially be split up into multiple files somehow.

This is, I'm sure, a much larger change, but at least a few people have asked for it (ex: #1008).