Add support for graphing multiple unrelated timeseries (metrics) in one graph

[juliusv]

It should be possible to correlate multiple metrics in the same graph. The graphing UI should show (dynamically) multiple expression input fields to enable that.

[brian-brazil]

The console templates graph library supports this, we'd have to hook it into /graph though.

[syepes]

+1

[smalot]

+1

[fmmr]

+1

[prasadus92]

+1

[alexellis]

~3 years on are we any closer to this happening? @brian-brazil

[juliusv]

@alexellis No, nobody has spent time on this yet as far as I know.

[mindfulmonk]

What about this query? It seems to work.

{__name__=~"node_filefd_allocated|node_filefd_maximum"}

[juliusv]

@mindfulmonk That works, but has the downside of giving you back all data in one expression, and especially if you apply any transformation that will drop the metric name (rate, increase, sum, avg, etc.), then you won't know anymore what is what.

[tomwilkie]

@davkal added a hack to the Cortex UI to allow multiple queries separated by semicolons. @juliusv do you think that might work here?

…

On Wed, 31 May 2017 at 15:50, Julius Volz ***@***.***> wrote: @mindfulmonk <https://github.com/mindfulmonk> That works, but has the downside of giving you back all data in one expression, and especially if you apply any transformation that will drop the metric name (rate, increase, sum, avg, etc.), then you won't know anymore what is what. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhQXG3gRkgAMhL3izXj6lIgc9zZJXks5r_X4lgaJpZM4AYIZW> .

[juliusv]

That really seems hacky to me (and could theoretically even conflict with future language changes). If we decide to do this in Prometheus itself at all, I'd prefer a full UI solution to this that allows you to add multiple input fields - either underneath each other or tabbed (like in OpenTSDB).

[G-Harmon]

+1 to this request

[davkal]

This is supported by Grafana's Explore, which is now available in grafana/grafana:latest as part of the Beta for the upcoming 6.0 release (and no longer behind a feature flag).

[codesome]

I was digging through issues for possible GSoC projects, so are we still keen on adding this into Prometheus?
/cc @juliusv

[juliusv]

I think yes, but it's probably not large enough for a GSoC project. Maybe a day or so...

[beorn7]

At this point, we should only consider this for the new ReactUI.

[cben]

There are two separate reasons this would be nice:

1. It's usually convenient to share horizontal axis — shifting/zooming time range in sync between all plots.

2. Overlaying different data on same plot and same vertical scale is sometimes desirable, sometimes confusing.

(1) is just UX convenience, you can simulate it now by copying same end time and range to all plots. But maybe it'd be acceptable to make this always the case — show the time range widgets only once for whole page — since when you want separate time scales you can always use multiple browser tabs?

For (2) there are several possible UI models:

• Hierarchical: a page is array of panels [Add panel] / [Remove panel]; each panel has array of expressions [Add expression] / [Remove expression].
• Flat: a page is array of expressions; between expressions have a checkbox [⭥Combine plots], rendering consecutive combined chunks into one plot. This may be weirder but I think it can be convenient to toggle between split and overlaid views.
  • Out of scope: An extreme variant of "flat" is to turn all expression entry boxes into a single editor, with plots popping between editor lines (e.g. CodeMirror can do it). Even weirder for new users 🤯, but would give power to re-arrange many plots by copy-paste...

[jsoref]

I'm trying to deal w/ the fact that alertmanager likes to send queries to prometheus of the form:

metric{something=...} > trigger_threshold

I'd really prefer alertmanager to be able to send this query to prometheus of the form:

super_impose_all_of_these{metric{something=...}, trigger_threshold}

the difference being w/ >, I can't see where the data was outside the clipped window.

[juliusv]

@jsoref I assume you are talking about the links back from Alertmanager to the generator of an alert (usually Prometheus), or generally being able to graph alert rule expressions with thresholds extracted somehow into their own visual indicator (vs. filtering).

Prometheus does not have a concept of a "threshold" per se, it just accepts arbitrary PromQL expressions in alerting rules, and those may or may not contain threshold-ish constructs in various places. So it's unfortunately not automatically possible to do such a thing in the Prometheus alerting model.

[jsoref]

@juliusv: yes, those. I'm not defining trigger_threshold as a magic concept for prometheus

super_impose_all_of_these{metric{something=...}, metric{something_else=...}} would be just as useful to me.

It's possible for some reason that I'd want the second line to have a slope, or a quadratic, or a cubic, or a logarithmic function instead of a constant value.

It just happens that for the average prometheus/alertmanager, what I want for that second metric is a constant value (=trigger_threshold).

Given a magical super_impose_all_of_these{...}, I could figure out how to get prometheus/alertmanager to generate links to prometheus/prometheus that used it and filled out super_impose_all_of_these{metric{something=...}, trigger_threshold} instead of metric{something=...} > trigger_threshold.

[roidelapluie]

If the labels are different you can use or

[jsoref]

@juliusv: so, I see this as having these pieces:

1. prometheus rules (which may contain something threshold like)
2. a link to alert manager
3. a link back to prometheus
4. prometheus offering a way to graph two things as i'm imagining them (which seems to be this issue)

If my magical super_impose_all_of_these{..., ...} existed (4), then (3) could be adjusted to use it. And (2) could be adjusted to use that. And worst case, (1) an extra field could be added to prometheus rules to specify an alternate query to use for graphing instead of the trigger query.

I don't see how or helps me. Certainly, I've tried using or, as in something like:

  topk(10, sum by (namespace, instance) (container_fs_usage_bytes{container!="",container!="POD"} / 1024 / 1024)) by(namespace, instance) or 500

and I get this message:

Error executing query: 1:3: parse error: set operator "or" not allowed in binary scalar expression

[roidelapluie]

You can try or vector(500)

[jsoref]

That works. (Could the error suggest that?)

[bboreham]

Discussed at bug-scrub: consensus was that projects such as Grafana and Perses cover this need, so we can close this issue.
Still open to a PR adding it.