OTEL enrichment agent do not set labels or host metadata during scaling event

[rupeshnemade]

We observed that whenever OTEL pod scales up we start seeing few logs with incomplete metadata (labels, host).
Expected

Actual
As we can see there are no labels & host present in Kuberenetes block

During scaling phase as OTEL pod doesn't have metadata from k8s API in its cache it sends logs without setting few metadata fields.
This behaviour is valid for short-lived pods but not appropriate for running pods.

Is this Kubernetes problem or this can be fixed in OTEL?

[swiatekm-sumo]

Since this only happens for new otel Pods, I have a suspicion that it's because the k8sprocessor processed data without first having populated its Pod cache completely. As I suggested on Slack, can you try adding a delay to the otel container (for example via initialDelaySeconds in the readiness probe) and see if that helps?

If this is the case, we can consider explicitly waiting for caches to sync while starting the processor.

[rupeshnemade]

I tried updating initialDelaySeconds in OTEL pods forwarding logs to Sumo from default 5sec to 20 sec but it didn't help to alleviate the issue.