Querying a distinct set of values of an attribute for dashboard variables

[utr1903]

Is your feature request related to a problem? Please describe.

In order to debug faster, it would be great if TraceQL could return distinct set of values of an attribute.

For example. Let's say I have 15 applications in my environment. I would like to create a generic dashboard for application monitoring and I would like to select one or more services from a Grafana dashboard variable to only see the necessary data belonging to these applications. You can think of any other attribute (e.g. k8s.cluster.name when I only want to consider the app instances running on specific clusters).

I was thinking that the select statement was thought for this purpose, yet it still gives the classic tabular output with Trace ID, Start Time, Name, Duration. I was wondering if this was a default Grafana visualization for Tempo and I wanted to check the response payload -> you can find it under the Additional context.

Describe the solution you'd like
It would be awesome, if the TraceQL could return distinct set of values so that one can create dashboard variables with them. Thereby, one could create a lot fewer dashboards to quickly troubleshoot anything.

{resource.service.name=~"$services" && k8s.cluster.name=~"$clusters"} | select(...)

Additional context
What the following query returns:

{} | select(span.trace.id)

{
    "traces": [
        {
            "traceID": "35a4b6118338d747a121326f2b3e1906",
            "rootServiceName": "shop-backend",
            "rootTraceName": "article-to-cart",
            "startTimeUnixNano": "1713423655080701384",
            "durationMs": 1031,
            "spanSet": {
                "spans": [
                    {
                        "spanID": "3d03a0a8ea7c1377",
                        "startTimeUnixNano": "1713423655080701384",
                        "durationNanos": "1031095620"
                    },
                    {
                        "spanID": "c03b7e126257ea18",
                        "startTimeUnixNano": "1713423655163670601",
                        "durationNanos": "192284284"
                    },
                    {
                        "spanID": "49de36b4b728b834",
                        "startTimeUnixNano": "1713423655159764475",
                        "durationNanos": "701149515"
                    }
                ],
                "matched": 10
            },
            "spanSets": [
                {
                    "spans": [
                        {
                            "spanID": "3d03a0a8ea7c1377",
                            "startTimeUnixNano": "1713423655080701384",
                            "durationNanos": "1031095620"
                        },
                        {
                            "spanID": "c03b7e126257ea18",
                            "startTimeUnixNano": "1713423655163670601",
                            "durationNanos": "192284284"
                        },
                        {
                            "spanID": "49de36b4b728b834",
                            "startTimeUnixNano": "1713423655159764475",
                            "durationNanos": "701149515"
                        }
                    ],
                    "matched": 10
                }
            ]
        },
        {
            "traceID": "c5be034179e1099e70d38b1a5f843762",
            "rootServiceName": "shop-backend",
            "rootTraceName": "update-payment",
            "startTimeUnixNano": "1713423654077661175",
            "durationMs": 744,
            "spanSet": {
                "spans": [
                    {
                        "spanID": "06da8150c9236cad",
                        "startTimeUnixNano": "1713423654077661175",
                        "durationNanos": "744340753"
                    },
                    {
                        "spanID": "83f7a8380fc8fc93",
                        "startTimeUnixNano": "1713423654133245134",
                        "durationNanos": "545627614"
                    },
                    {
                        "spanID": "f40dcad04d6a8fef",
                        "startTimeUnixNano": "1713423654172270340",
                        "durationNanos": "313660026"
                    }
                ],
                "matched": 3
            },
            "spanSets": [
                {
                    "spans": [
                        {
                            "spanID": "06da8150c9236cad",
                            "startTimeUnixNano": "1713423654077661175",
                            "durationNanos": "744340753"
                        },
                        {
                            "spanID": "83f7a8380fc8fc93",
                            "startTimeUnixNano": "1713423654133245134",
                            "durationNanos": "545627614"
                        },
                        {
                            "spanID": "f40dcad04d6a8fef",
                            "startTimeUnixNano": "1713423654172270340",
                            "durationNanos": "313660026"
                        }
                    ],
                    "matched": 3
                }
            ]
        }
    ],
    "metrics": {
        "inspectedBytes": "86932",
        "totalBlocks": 1,
        "completedJobs": 1,
        "totalJobs": 2,
        "totalBlockBytes": "295526"
    }
}

[joe-elliott]

So Tempo can return these distinct values using the tag names and tag values endpoints.

Grafana supports this now by choosing a Tempo datasource when creating a dashboard variable:

[utr1903]

@joe-elliott Thanks a lot for your super fast response chief!

I was testing it per this example which uses the latest Tempo but not Grafana. The Query type field selector was missing there. After switching to the latest Grafana version, it looks wonderful!

Shall I create a PR for the example to use the latest version of Grafana?

[joe-elliott]

Shall I create a PR for the example to use the latest version of Grafana?

If you have the time, that would be great.