Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add {job="karpenter"} to "label_values" to discriminate from other controllers #6163

Open
yogeek opened this issue May 7, 2024 · 3 comments · May be fixed by #6181
Open

add {job="karpenter"} to "label_values" to discriminate from other controllers #6163

yogeek opened this issue May 7, 2024 · 3 comments · May be fixed by #6181
Assignees
Labels
documentation Improvements or additions to documentation good-first-issue Good for newcomers

Comments

@yogeek
Copy link

yogeek commented May 7, 2024

In some dashboards, (e.g. https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/preview/getting-started/getting-started-with-karpenter/karpenter-controllers.json#L354C37-L354C78) filters are defined like this :

"query": "label_values(controller_runtime_reconcile_errors_total,controller)",

but in a cluster where other controller metrics have a "controller" label (like aws-load-balancer-controller), the filter does not only include karpenter controllers.

adding the job would allow to avoid getting other "non-karpenter" controllers :

"query": "label_values(controller_runtime_reconcile_errors_total{job=\"karpenter\"},controller)",

If confirmed, I'll make a PR to fixit

@jonathan-innis
Copy link
Contributor

Yeah, it looks like the other metrics in the dashboard do this correctly (see https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/preview/getting-started/getting-started-with-karpenter/karpenter-controllers.json#L292). Can you copy what's present in the other controller_runtime metrics in that file and just apply that to this metric. That should allow us to select the controller dynamically in the dashboard.

@jonathan-innis jonathan-innis self-assigned this May 9, 2024
@jonathan-innis jonathan-innis added the documentation Improvements or additions to documentation label May 9, 2024
@jonathan-innis
Copy link
Contributor

@yogeek Is this something that you want to be assigned to if you're going to open the PR?

@yogeek
Copy link
Author

yogeek commented May 11, 2024

@jonathan-innis I think there is a misunderstanding : I think the issue is not on the metrics (which are using the "controller" variable as they should, like in the one you pointed out like : METRIC{controller=\"$controller\"}) but in how this "controller" variable's values are retrieved : using label_values(controller_runtime_reconcile_errors_total, controller) implies that all the controllers that expose the controller_runtime_reconcile_errors_total metric (karpenter is not the only one, aws-load-balancer-controller is also exposing it for example, and maybe other controllers as it is a quite common metric for a controller) are retrieved and listed in the "controller" variable's values, hence appearing in the drop-down variable filter.
However, the only controllers needed for the karpenter dashboard are karpenter's ones so we have to find a way to select only karpenter's label_values, and using {job="karpenter"} is one way to do.

I prepared a PR here #6181

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation good-first-issue Good for newcomers
Projects
None yet
2 participants