Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework dashboard #151

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Rework dashboard #151

wants to merge 1 commit into from

Conversation

SuperQ
Copy link
Owner

@SuperQ SuperQ commented Apr 23, 2024

Rework the dashbaord to be more useful.

  • Use rate() where needed to get correct results.
  • Add support for native histograms.
  • Improve dashboard variables.

Rework the dashbaord to be more useful.
* Use `rate()` where needed to get correct results.
* Add support for native histograms.
* Improve dashboard variables.

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

Fixes: #150

@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

Fixes: #100

@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

Fixes: #90

@dominikh
Copy link

Add support for native histograms.

This seems to be breaking the dashboard for people who aren't using native histograms. I'm getting this for the Average Latency graph:

Status: 500. Message: bad_data: invalid parameter "query": 1:1: parse error: unknown function with name "histogram_avg"

The new dashboard doesn't seem to break out multiple ping targets into their own panels anymore. This was useful to compare hosts and check if they behaved differently, e.g. due to routing. Being able to look at the sum of all hosts (by setting host and ip to all) is definitely useful, though.

And I can see how breaking them out would be bad if someone had dozens of targets. I'm not well-versed in Grafana; is there a way to add a checkbox that toggles this behavior?

@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

What version of Prometheus do you have?

@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

I can add the row configuration back in.

@dominikh
Copy link

What version of Prometheus do you have?

I'm on version 2.47.2. histogram_avg seems to have been added in 2.51.0, which only released in March 2024. Even then, the function is documented as

This function only acts on native histograms, which are an experimental feature.

and most users probably have their data in classic histograms, not native ones.

@SuperQ
Copy link
Owner Author

SuperQ commented Apr 23, 2024

Yes, and that's why there is an or in the query now. If the native histogram doesn't return data, it will use the classic histogram data.

@bboehmke
Copy link

Hi,

I also checked the reworked dashboard and it looks like the or is missing in the 3rd panel Average Latency.
This results in no data for if native histograms are not enabled.

@@ -472,19 +454,19 @@
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "smokeping_response_duration_seconds_sum{host=\"${target:raw}\"} / smokeping_response_duration_seconds_count{host=\"${target:raw}\"}",
"legendFormat": "{{host}}",
"expr": "histogram_avg(rate(smokeping_response_duration_seconds{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]))",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should look like this

Suggested change
"expr": "histogram_avg(rate(smokeping_response_duration_seconds{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]))",
"expr": "histogram_avg(rate(smokeping_response_duration_seconds{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]))\nor\nrate(smokeping_response_duration_seconds_sum{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]) / rate(smokeping_response_duration_seconds_count{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval])",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants