Replies: 1 comment 1 reply
-
What do you mean by |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I use celery to complete ETL tasks. To do so, I create a script that creates one task per item in our database, but our database contains millions of items, so I don't want to create millions of tasks all at once. Ideally, I'd like my ETL script to monitor celery and keep the workers fed at roughly the rate they consume tasks.
To do this, I need a way to figure out the number of tasks awaiting workers. I've been unable to figure this out for years, and there are some solutions around, but they fall short:
inspect
will tell you the number of reserved tasks or the number of active tasks, but it won't tell you the number of "enquequed" tasks.As far as I can tell, there's no way to do this, but I'm flummoxed how that could be because workers know how to grab tasks from the queue that need processing.
So my questions are:
Is there a way to identify the number of tasks that are ready for workers?
If that's not easy: How can workers distinguish which tasks are failed vs which need to be processed?
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions