Pagination for tag search endpoint

[gng0]

We've been encountering an issue with 500 error responses when making requests to the /api/v2/search/tag/name/values endpoint. This is due to the response size exceeding the maximum configured value, as outlined in the Grafana documentation here: https://grafana.com/docs/tempo/latest/troubleshooting/search-tag/

Although the documentation suggests reducing the cardinality of tags or increasing the max_bytes_per_tag_values_query parameter, we don't think these solutions are feasible in the long-run.

We'd like to suggest pagination on this API endpoint, which will resolve the current issue and be a more scalable solution. It would also lead to better user experience as the data load times would be able to be reduced.

[joe-elliott]

pagination is difficult, but likely not impossible. i'd have to think on it some.

we have recently added a streaming tags endpoint that might meet your needs. it can be exercised using the tempo-cli tool (or any appropriate grpc client):

https://github.com/grafana/tempo/blob/main/cmd/tempo-cli/cmd-query-search-tag-values.go
#3460

this will be released in 2.5.

[gng0]

Hi @joe-elliott - sorry I think my initial comment might have been a bit misleading, as we aren't directly calling that API endpoint. The endpoint is being hit from the “Search” tab in the default Grafana Tempo plugin. So we are seeing 500's every time someone loads that page in “Explore” mode.

Will the new streaming endpoint you mentioned be used in the Grafana Tempo plugin UI?

[joe-elliott]

Will the new streaming endpoint you mentioned be used in the Grafana Tempo plugin UI?

The Grafana team is aware of this endpoint, but has no immediate plans to implement it. Since you are clearly showing interest do you want to create an issue on https://github.com/grafana/grafana/? I can +1 the issue and make sure it's tagged appropriately.

[jcmackie]

Hello @joe-elliott ,
Thanks for the response.

I was going to ask you if the multiplexed http/grpc issues with TraceQL streaming is resolved, but then I searched and saw your PR: #3300

Before I start playing around with our dev environment, did the above PR make it possible to multiplex http/grpc connections over an AWS ALB?

[joe-elliott]

Before I start playing around with our dev environment, did the above PR make it possible to multiplex http/grpc connections over an AWS ALB?

I'm honestly not sure, but I don't see why not. You'd need to forward http2/http to the same port in Tempo.