Record pod Ready status, or all pod status conditions

[gdvalle]

Component(s)

receiver/k8scluster

Is your feature request related to a problem? Please describe.

I would like to be able to see a pod's Ready status in metrics.

Describe the solution you'd like

A named metric with an analog to k8s.container.ready ,like k8s.pod.ready.

Or, a more generic solution where all pod status conditions are represented, i.e. k8s.pod.status.condition. I might prefer this as it captures more states.

I would like to include attributes for the detail info fields (message, reason), for either approach.

Describe alternatives you've considered

Using the existing k8s.container.ready metric. It's unfortunately not complete because of pod ReadinessGates.

Additional context

See #32933 for an impl of k8s.pod.ready, currently sans attributes.

If we can agree, I think the condition status metric makes more sense. I think we need agreement on how to represent it: other things in this receiver (e.g. k8s.pod.phase) use values to represent states, so 0=false, 1=true, 2=unknown could fit, but the ergonomics are a bit poor.

[github-actions]

Pinging code owners:

• receiver/k8scluster: @dmitryax @TylerHelmuth @povilasv

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

This feels reasonable, especially if there is existing precedence in KSM.

Is PodStatus already watchable via the k8sobjectsreceiver?

[gdvalle]

Is PodStatus already watchable via the k8sobjectsreceiver?

Do you mean k8sclusterreceiver? It has existing watch support for Pod, which should cover the Status struct AFAIK.

k8sobjectsreceiver, I'm not certain, but glancing at the field selection language I'm not sure how you would get the ready condition out of the list.

[povilasv]

I think given we have k8s.container.ready we could add k8s.pod.ready.

Regarding pod.status.condition does Kube State Metrics have it? Would be interesting to see what they do around that and what kind of values are there.

[sirianni]

I think we need agreement on how to represent it: other things in this receiver (e.g. k8s.pod.phase) use values to represent states, so 0=false, 1=true, 2=unknown could fit, but the ergonomics are a bit poor.

Please no.

See #29157 for a generic k8s.pod.state metric modeled after kube-state-metrics.

[sirianni]

In general, I feel like the piecemeal and fragmented approach to Kubernetes metrics in OTel is going to make it impossible to have any vendors build meaningful user experiences on top of this dataset, ultimately delaying any possible standardization in this space.

[povilasv]

In this case k8s.pod.state sounds to me more like resource attribute than a metric? Then IMO we can add is a resource attribute.

Regarding:

think we need agreement on how to represent it: other things in this receiver (e.g. k8s.pod.phase) use values to represent states, so 0=false, 1=true, 2=unknown could fit, but the ergonomics are a bit poor.

I guess your against this enum approach? IMO then this should be added to the spec, how to correctly represent enums in otel, so we can solve it once and for all.

[TylerHelmuth]

In general, I feel like the piecemeal and fragmented approach to Kubernetes metrics in OTel is going to make it impossible to have any vendors build meaningful user experiences on top of this dataset, ultimately delaying any possible standardization in this space.

We have had an uptick in k8s components feature request, it would be be very nice to make progress on open-telemetry/semantic-conventions#1032

[sirianni]

Thanks @TylerHelmuth . In general, my team has been happy with the metrics collected by kubeletstatsreceiver and how they are modeled. They are struggling significantly with the "state" metrics that come from k8sclusterreceiver. We are coming from a Datadog background.