Using Celery in a critical production environment

[lfse-slafleur]

Hi everyone!

Currently we are evaluating Celery to see if we can use this technology as our task queue framework. However, we are in a situation where it is critical that tasks are performed but also results are delivered. One of the main points that I have been unable to answer with the documentation is how to proceed when the Celery app/client is restarted and the references to any outstanding tasks are lost. How can you retrieve the result once the Celery app is restarted?

Steps:

1. Celery app (with amqp broker & rpc backend) sends a long running task 'task1' to the workers.
2. Celery app reboots due to an update or crash. The 'AsyncResult' reference is lost of course.
3. Celery app restarts fresh but has no clue which tasks are completed but still need to be .get()

What have I researched myself:

1. I have attempted to use inspect but you may only retrieve .reserved(), .scheduled() and .active(). Completed is not an option.
2. After step 3, construct an AsyncResult('<taskid>') and use .get(). Note that it blocks indefinitely. Wireshark shows it is trying to retrieve the result from some random queue.
3. Try app.control.inspect().query_task('<taskid>') but it only shows meta information and nothing about the result / backend.

So none of the options I have researched have worked yet. How would we go about picking up all the existing tasks with a fresh Celery app/client instance? Or is the paradigm to purge/kill everything and start all tasks fresh on a reboot of the app?

Thanks in advance!

[lfse-slafleur]

Answered my own question!

You can use AsyncResult('<taskid>') but only when you use the database type backend. This is also stated here in the docs: https://docs.celeryq.dev/en/latest/userguide/tasks.html#rpc-result-backend-rabbitmq-qpid

The RPC result backend (rpc://) is special as it doesn’t actually store the states, but rather sends them as messages. This is an important difference as it means that a result can only be retrieved once, and only by the client that initiated the task. Two different processes can’t wait for the same result.