[Feature Request] [Prometheus]: Remove limits for metrics browser

[erin-noe-payne]

What happened:

The prometheus metrics browser currently has a hardcoded limit of 10k labels and 50k values. Our prometheus deployment has ~160k metrics. In practice, this means that the metrics browser cannot reliably be used to build a query, since > 2/3rds of the metrics cannot be viewed or selected from the metrics browser.

This also leads to inconsistent behavior between the metrics browser and the query editor, since #39363 means that the PromQL Query Editor no longer limits suggestions.

In the video below you can see that I have a bunch of metrics named zookeeper_*. From the PromQL Query Editor I am able to write a query against these metrics, but from the metric browser, I cannot. The metric browser truncates the metrics list at 50k, which for our metrics is still in the "a"s.

13-21-4pybe-v2ve6.mov

What you expected to happen:

The metrics browser allows me to build a query against my existing metrics, labels, and values.

How to reproduce it (as minimally and precisely as possible):

• Connect to a prometheus datasource with > 50k metrics.
• Open the metrics browser, attempt to select a metric past the 50k truncation point.

Anything else we need to know?:

• As I see it, the core concern here is performance. However, the current solution limits functionality and I don't believe it helps performance.
  • The prometheus api doesn't support pagination or querying based on a search string - it just dumps the complete list of all values (see docs for querying labels, docs for querying values). A deployment with really high cardinality is going to pay the cost of downloading, json parsing, allocating an array, etc - the current limits don't do anything to protect us.
  • The metrics and values are rendered into a react-window FixedSizeList, so they are virtualized in the DOM. Only labels are not virtualized.
• I'm happy to open a PR with a fix. In our fork, I simply removed the code that slices the list (see here). If this is an acceptable approach I can contribute it back. If there are other suggestions I would take that feedback and open a PR.

Environment:

• Grafana version: 8.2.1
• Data source type & version: Prometheus 5.0.0
• OS Grafana is installed on: Mac OS 11.6
• User OS & Browser: Mac OS 11.6 / Chrome
• Grafana plugins:
• Others:

[ivanahuckova]

Thank you for creating the issue, I am moving it to discussions where we are collecting all feature requests and discussions.

[erin-noe-payne]

Ah, sorry! I'll go through the feature requests flow in the future.

[eglizzy]

@ivanahuckova How do we move something like this forward? We can open a PR if that helps.

[ivanahuckova]

Hello @autoric and @eglizzy there are actually 2 open external PRs for this:

• prometheus metrics browser: remove autocompletion limits #40530 by @edma2
• Prometheus: Remove limits on metrics, labels, and values in Metrics Browser #40660 by @autoric

@edma2 and @autoric as your PRs are duplicated. To move forward with any of those, it would be awesome if you did and shared some performance testing for 500k/1M+ metrics. Also, both of those PRs removes autocomplete limit for other labels and values which I think might be problematic, but maybe I am wrong. If you want to remove limits also on this, could you do performance testing for those as well. Otherwise would you change your solution so it only affects metrics.

[ivanahuckova]

Implemented in #40660