[Feature] Standalone Elastic Agent Composite Agent Policies

[BenB196]

Proposal

Support defining multiple configRef secretName values to build a policy from multiple secrets.

Use case. Why is this important?

I've manage some fairly large system generated Elastic Agent policies for Standalone agents. I've recently been hitting an issue where the policies have grown past Kubernetes Secret Size limit, 1 MiB, and have had to do some hacky stuff to try and bring the size down.

Ideally, I should be able to provide a number of secrets that the Elastic Agent/ECK operator can merge together to form a singular policy file for the agent when deployed, allowing for configs >1 MiB.

[pebrc]

We would need a different way of mounting configuration into the Pods to make this proposal work. If the operator just collates the configuration from mulitple source secrets into one that is then mounted into the Pods we would run into the limitation again.

One way to work around this would require the configuration to be written and mounted into a custom volume in the Pod e.g. using a projected volume

Another way would be to sidestep k8s alltogether and use the Elastic Stack terraform provider (does not work for your use case as you are using Agent standalone)

[BenB196]

Yeah, there isn't a "clean" way of doing this today. Another way I've thought about achieving this, was to use an initContainer + shared emptyDir pattern. Where you'd mount multiple secrets to an initContainer, then use something like yq to merge the files into one config, store that on a shared emptyDir, and then have the main Elastic Agent use that generated config instead.

Unfortunately, for us, Elastic Stack terraform provider currently doesn't work as we're mainly dealing with Synthetics (ICMP/TCP) configs, which have deprecated managed integration configs directly in favor of "Private locations". I did notice that Elastic (Kibana) recently added public API support via: elastic/kibana#169547, so I've gone ahead and opened elastic/terraform-provider-elasticstack#610, as maybe that is a "better" long-term solution.