receiving celery_worker_tasks_active = 0

[maxgacrux]

celery_worker_tasks_active{hostname="worker2"} 0.0
celery_worker_tasks_active{hostname="worker1"} 0.0

Hey! i get 0 active tasks even though there are currently 10. it seems to me that celery-exporter monitoring is not updating active tasks often enough

[danihodovic]

Have you enabled worker_send_task_events in your app?

https://docs.celeryq.dev/en/stable/userguide/configuration.html#worker-send-task-events

[maxgacrux]

@danihodovic Hey! thank you for the quick reply. yes, I have included in the application what you say. but I still don't get what I want. i use fastapi with celery

celery.conf.task_send_sent_event = True

[maxgacrux]

@danihodovic maybe the problem is that I'm using workers with gpu?

command: celery worker --app=app.worker.celery -P solo --loglevel=info -Q main_task -n worker1

[danihodovic]

I don't think that should be a problem. Increase the log level of the celery-worker and see if it's logging anything interesting.

[maxgacrux]

@danihodovic how to enlarge the magazine?

[danihodovic]

https://github.com/danihodovic/celery-exporter/blob/master/src/cli.py#L56

Either export an environment variable

export CE_LOG_LEVEL=DEBUG
./celery-exporter

or use a command line argument

./celery-exporter --log-level=DEBUG

[maxgacrux]

@danihodovic I turned on the debug mode and made sure that all metrics are sent, except for task activity

[maxgacrux]

@danihodovic I also noticed that from time to time celery-exporter gives out non-existent hostnames. here is an example:

celery_task_started_created{hostname="435ab5a1d257",name="create_task"} 1.667807616992973e+09
celery_task_started_created{hostname="worker",name="create_task"} 1.6678076170305264e+09

even though I gave my worker a name:

command: celery worker --app=worker.celery --loglevel=info -Q default -n worker

[danihodovic]

Are you sure you're running a single worker? The exporter reads the hostname from the celery events.

[maxgacrux]

@danihodovic yes, I am absolutely sure of it. above is the command i use in my docker-compose.yml

[maxgacrux]

@danihodovic the only moment I see now is the launch of flower. can there be problems because of this?

command: flower --app=worker.celery --port=5555 --broker=pyamqp://guest:guest@rabbitmq:5672/ --broker_api=http://guest:guest@rabbitmq:15672/api/vhost

[danihodovic]

I'm not running flower so it's hard to tell

[kosyan62]

Hello there!
I met the same problem and spent some time analyzing it. It seems that the only place where
celery_worker_tasks_active updating is on the worker_heartbeat event from celery. @maxgacrux do you use --without heartbeat? Disabling it (enabling heartbeat for worker) should do the trick.