Tempo service graphs: make prefix of metrics configurable

[kvrhdn]

The Tempo service graphs panel currently uses the metrics generated by the Grafana Agent. The Agent generates the following metrics related to service graphs:

traces_service_graph_request_client_seconds_bucket
traces_service_graph_request_client_seconds_count
traces_service_graph_request_client_seconds_sum
traces_service_graph_request_server_seconds_bucket
traces_service_graph_request_server_seconds_count
traces_service_graph_request_server_seconds_sum
traces_service_graph_request_total
traces_service_graph_unpaired_spans_total
traces_service_graph_request_failed_total
traces_service_graph_dropped_spans_total

With Tempo we want to generate the same metrics and use the same panel in Grafana, but the metric names will likely be slightly different.
Our design proposal describes our implementation of this processor here: https://github.com/kvrhdn/tempo/blob/kvrhdn/metrics-generator-proposal/docs/design-proposals/2022-01%20Metrics-generator.md#service-graph

I can imagine we might opt to generate metrics like tempo_service_graph_request_total and maybe even grafanacloud_traces_service_graph_request_total.

Would it possible to make the prefix configurable? I.e. Grafana uses metrics like <prefix>_service_graph_request_total in which <prefix> defaults to traces and can be configured in the UI.

[connorlindsey]

Could you explain a bit more why the metric names might differ? Are there any technical constraints? If possible, it'd be nice to keep them consistent.

[kvrhdn]

The main reason to pick a different metric name is to avoid collision with metrics generated by the Agent and Tempo. If there are two sources of service graphs metrics (for instance if both the Agent and Tempo are enabled) they might be generating the same metrics. This can result in skewed data if they process the same traces (i.e. amount of calls will be counted double).

But thinking about this some more... an alternative here would be to just add a label to the metrics, for instance source="agent" and source="tempo". That way you could easily filter out either of them (or use both at the same time). So this doesn't really justify it 🤔

[connorlindsey]

I like the idea of just adding a label

[kvrhdn]

We had a quick discussion about this internally and figured it will be best to keep the metrics exactly the same. Labels will be an easy way to differentiate between the metrics from the Agent and Tempo.

Some reasons:

• It will be easier overall to use the same names as you don't have to configure anything special.
• We optimise reusability of queries/dashboards between the Agent and Tempo.
• Users are unlikely to generate metrics from Agent and Tempo at the same time (or this would be temporary).

So we can close this feature request as no work is needed 🙂

[kvrhdn]

Hmm, should I just leave this discussion as-is? There is no close button for discussions. I can lock it but I don't want to block other people from leaving comments.

[connorlindsey]

Added the 'wontfix' label and will lock as well.