Celery worker silently stops working in some conditions under high load

[fruttasecca]

Describe the bug

After a period of very high resource contention the celery worker stopped processing tasks. The probe and liveness checks did not lead to a restart of the pod, which is what fixed the issue. We definitively want the probes to notice the problem and restart the pod in case this happens, moreover, some tuning related to settings like the CELERY_BROKER_HEARTBEAT could be done to reduce load.

Environment

Happened in the cloud where our smaller instances are on 2 vcpus, but could happen anywhere of course.