Metrics with multiple workers not properly working?

[ayyuno]

hi! first of all, thanks for providing this tool to us.

The metrics page looks good unless we use multiple workers (we use gunicorn to start 4 workers for an API). On every request the metrics page is served by a different (and mostly unpredictable) worker, which returns the metrics of this specific worker only.

As an example, this leads to "jumping" counter metrics, making them mostly unusable in Prometheus/Grafana.

We use a simple config for the requests metrics:

instrumentator.add(metrics.requests())

And then access the the /metrics endpoint (for simplification I focus on a single metric only) ...:

#1 ... http_requests_total{handler="/credits/",method="GET",status="200"} 161.0
#2 ... http_requests_total{handler="/credits/",method="GET",status="200"} 223.0
#3 ... http_requests_total{handler="/credits/",method="GET",status="200"} 197.0
#4 ... http_requests_total{handler="/credits/",method="GET",status="200"} 161.0
#5 ... http_requests_total{handler="/credits/",method="GET",status="200"} 161.0
...

Request #1 and #4 seem to be served by the same worker. So even if there's no traffic on the page, the resulting graph using the rate function would look like a white noise signal:

I was already thinking about including the worker ID and then group the results. But that's not very reliable at all. Did you see this before or do we miss something?

greetings, Hans

[trallnag]

Hey Hans, I'll take a look at it this evening and provide a proper solution / answer. The issue should be one of the following:

1. I forgot to set the correct multiprocess mode in the metrics.py file where I defined the "instrumentation handlers" that contain the Prometheus metrics.
2. You forgot to expose the /metrics endpoint with taking multiprocess mode into consideration. Take a look at the docs for the expose() method or create your own endpoint that serves metrics.
3. A combination of point 1 and 2.

[ayyuno]

Hi Tim, I presume the fault is on our side. We missed the hint on the multiproc mentioned in the automatic doc, function expose. It's mentioned in the raise condition (If prometheus_multiproc_dir env var is found ... ).

THANKS for the tip!

Together with https://github.com/prometheus/client_python#multiprocess-mode-gunicorn we were able to reconfigure the API. Looks good now, but we will need some more investigation.

best Hans

[nielstenboom]

Wow thanks a lot! I also got stuck on this today for quite some time haha.

Would it be an idea to add a section to the README regarding this? I think it could prevent quite a few people making the same mistake :).

Because as soon as you'll containerize fastapi in the default way then you'll run into this problem (https://hub.docker.com/r/tiangolo/uvicorn-gunicorn-fastapi)

[trallnag]

Good point #26