Retrieving results across multiple queues #8756
Unanswered
sdrabblescripta
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have two queues although only one is actively processing tasks, call them A and B. Each is deployed in an individual docker deployment consisting of an app container and a celery worker. Both queues use the same rabbit MQ running in its own container. To clarify, there is exactly one rabbit mq container shared by all workers.
I can successfully start tasks in B.celery from A.app. I can watch the task in B's log and see it completes. In B.app I am able to retrieve the task's state and result.
If, however, I try to retrieve the task's result/ state in A.app, those are always empty and PENDING, and
get()
/wait()
just hang.This is problematic because A is where, for the most part, all tasks are started, and A needs to send the same task to each of B, C, D, ... then wait for all tasks to complete before moving on. I can't use chained tasks or similar because the tasks all need to run in parallel.
Is what I'm doing just not possible? If not, how would I go about obtaining the results in A for all tasks run in other queues?
Beta Was this translation helpful? Give feedback.
All reactions