You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I initiate the tasks, Celery successfully picks up three tasks and processes them. However, once one of the workers completes its task, Celery fails to spawn a new worker to pick up another task from the broker queue. It only does so when the third worker completes its task.
This behavior is not optimal for me as I would prefer tasks to be initiated as soon as a worker becomes available.
Could this be a bug, or am I misunderstanding some concepts? I thought that worker_prefetch_multiplier would address this issue.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm utilizing Celery 5.4.0 with a RabbitMQ broker.
I have a list of tasks that are expected to run for extended periods, and I have three workers available for processing.
These are the configurations my Celery app is using:
When I initiate the tasks, Celery successfully picks up three tasks and processes them. However, once one of the workers completes its task, Celery fails to spawn a new worker to pick up another task from the broker queue. It only does so when the third worker completes its task.
This behavior is not optimal for me as I would prefer tasks to be initiated as soon as a worker becomes available.
Could this be a bug, or am I misunderstanding some concepts? I thought that
worker_prefetch_multiplier
would address this issue.Beta Was this translation helpful? Give feedback.
All reactions