api/v1/metadata and api/v1/label/__name__/values returning different values

[gburiola]

What did you do?

I have a prometheus target with hundreds of thousands of metrics (MY_TARGET in the config below). I do not have control over the metrics being exposed and I only care about a handful of metrics that contain label MY_LABEL

I created the metric_relabel_configs below to drop all metrics that do not contain a specific label (MY_LABEL).

scrape_configs:
- job_name: MY_TARGET
...
  metric_relabel_configs:
  - source_labels: [MY_LABEL]
    separator: ;
    regex: ^$
    replacement: $1
    action: drop

The config above works and the metrics I don't care about are not available when I call api/v1/query.

When I open https://PROMETHEUS_URL/graph with Enable autocomplete checked, it makes 2 calls to prometheus API

1. https://PROMETHEUS_URL/api/v1/label/__name__/values
2. https://PROMETHEUS_URL/api/v1/metadata

The first call correctly returns a small list containing the names of only the metrics I care about.

The second (v1/metadata), returns all metrics, including those that have been dropped by the config above.

What did you expect to see?

I would have expected v1/metadata to return the same list as api/v1/label/__name__/values but instead if returns hundreds of thousands more. All of the ones that have been dropped by the config above.

Is it possible that api/v1/metadata is being populated BEFORE the metrics are being dropped by the config above?

What did you see instead? Under which circumstances?

.

System information

Linux 4.18.0-372.78.1.el8_6.x86_64 x86_64

Prometheus version

prometheus, version 2.49.1 (branch: HEAD, revision: 43e14844a33b65e2a396e3944272af8b3a494071)
  build user:       root@6d5f4c649d25
  build date:       20240115-16:58:43
  go version:       go1.21.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[bboreham]

Is it possible that api/v1/metadata is being populated BEFORE the metrics are being dropped by the config above?

Yes. Metadata is associated with a "metric family" (just the name of the metric) while drop is at the level of individual metrics.
Also the parser works from top to bottom and usually the metadata is before the metrics.

It's reasonable as a feature request: "if all series for a family are dropped, ignore the metadata", but not easy to implement.