Queue length is wrong when using Redis and task priorities

[ArchFrosty]

tldr; only the priority 0 queue is counted

This one is fairly simple:
when you get active queues here

celery-exporter/src/exporter.py

Line 229 in c837bab

queues = self.app.control.inspect().active_queues() or {}

Celery will indeed return you the active queues from its point of view - for example the default queue 'celery'. The problem is that when using Redis, priority is implemented by actually crating multiple queues in Redis.
By default it creates 3 additional queues: 'celery\x06\x163' 'celery\x06\x166' 'celery\x06\x169' - queues 3 6 and 9 with '\x06\x16' as separator between name and priority number. Priority numbers used when submitting tasks are then mapped to these queues (0 highest prio and 9 lowest), the highest priority queue (with priority value 0) is just called its name, in this case 'celery' without separator and number

So then when you eventually call

celery-exporter/src/exporter.py

Line 250 in c837bab

length = queue_length(transport, connection, queue)

and eventually call llen

celery-exporter/src/exporter.py

Lines 449 to 450 in c837bab

	def redis_queue_length(connection, queue: str) -> int:
	return connection.default_channel.client.llen(queue)

on the name of this queue, you only get the tasks in the priority 0 queue.

Also the separator and then number of queues is configurable.
https://docs.celeryq.dev/en/stable/userguide/routing.html#redis-message-priorities

[danihodovic]

#204 (comment)

tl;dr I don't think I will add queue length for Redis.

[zdenekbauer]

Hi, we have the same problem. I saw that there was already prepared pull-request for that, can you please consider it again? It would be amazing to have this feature.

[danihodovic]

@zdenekbauer I will not be adding it for now. Feel free to fork the repo.

See #204 (comment)