Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056

Emilianissimo · 2023-02-06T17:33:46Z

Checklist

I have verified that the issue exists against the main branch of Celery.
This has already been asked to the discussions forum first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the main branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the main branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.7

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.16
Minimal Celery Version: 5.2.7
Minimal Kombu Version: 5.2.4
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Django==2.2.28
celery==5.2.7
billiard==3.6.4
pytz==2022.7.1

Other Dependencies

N/A

Minimally Reproducible Test Case

# dajngo-test-case
    def simple_test_test(self):
        mocked_smth = self.patch(
            'path.to.tasks.patched_counter',
            side_effect=[Exception, True]
        )
        with self.assertRaises(Retry):
            patching_method()
        self.assertEqual(mocked_smth.call_count, 2)

# tasks.py
@shared_task(bind=True)
def patching_method(self):
    try:
        patched_counter()
    except Exception as e:
        raise self.retry(exc=e) # Retry raises, but do not retries task as well

def patched_counter():
    pass

Expected Behavior

I'm expecting, that after raising Retry inside Django tests it will retry task and set patched_counter to 2

Actual Behavior

So it is just running trough first side_effect, raises Retry, so assertRaises works, and going through test function without retry task.
My suggest is this is in case of:
a) Retry just cannot be faster, then test case (it is not blocking, so it is async maybe?)
b) Retry just do not running (I'm not sure why?)

It worked before update to 5.2.7 from 4.3.1, also updated billiard and pytz to required.

Also here is the ENVs:

CELERY_WORKER_DISABLE_RATE_LIMITS = True
CELERY_SEND_TASK_ERROR_EMAILS = False
CELERY_TASK_ANNOTATIONS = {
    "*": {
        "on_failure": celery_on_failure # incapsulated logging
    }
}
CELERY_QUEUE_HA_POLICY = 'all'
BROKER_HOST = os.getenv('BROKER_HOST', 'localhost')
BROKER_PORT = os.getenv('BROKER_PORT', 5672)
BROKER_USER = os.getenv('BROKER_USER', 'guest')
BROKER_PASSWORD = os.getenv('BROKER_PASSWORD', 'guest')
BROKER_VHOST = os.getenv('BROKER_VHOST', '/')

CELERY_RESULT_BACKEND = 'rpc://{BROKER_USER}:{BROKER_PASSWORD}@{BROKER_HOST}:{BROKER_PORT}/{BROKER_VHOST}'.format(
    BROKER_USER=BROKER_USER,
    BROKER_PASSWORD=BROKER_PASSWORD,
    BROKER_HOST=BROKER_HOST,
    BROKER_PORT=BROKER_PORT,
    BROKER_VHOST=BROKER_VHOST,
)
CELERY_RESULT_PERSISTENT = True
CELERY_TASK_IGNORE_RESULT = True

And especially for testing:

CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True

Tried to run with eager_mode to False, but it says that Retry does not raising.

Here also the celery_app file:

from django.conf import settings

project_name = 'name' 
app = celery.Celery(project_name)
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

The text was updated successfully, but these errors were encountered:

open-collective-bot · 2023-02-06T17:33:48Z

Hey @Emilianissimo 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

andrewdieken · 2023-07-07T21:50:47Z

I'm running into, what I believe to be, a similar issue. I have a Django v3.2.19 application currently running Celery v4.3.0 and am attempting to upgrade to v5.2.7. After upgrading, a few test cases around task retry logic started failing which I believe to be related to this PR. Below is an example showcasing the issue

class SomeException(Exception):
    """ Custom Exception Class """
    pass


def some_function(number):
    if number < 0:
        raise SomeException
    return True


@app.task(bind=True)
def some_task(self):
    try:
        random_number = random.randint(0, 100)
        some_function(random_number)
    except SomeException as e:
        retry_countdown = int(random.uniform(2, 4) ** self.request.retries)
        raise self.retry(countdown=retry_countdown, max_retries=5)


class CeleryRetryTestCase(TestCase):

    @mock.patch('some_function')
    def test_celery_retry(self, mocked_some_function):
        retryable_exceptions = [
            SomeException(),
        ]
        mocked_some_function.side_effect = retryable_exceptions * 10
        with self.assertRaises(MaxRetriesExceededError):
            some_task.delay()

        # we should have called some_function() 6 times
        self.assertEqual(mocked_some_function.call_count, 6)

Running the CeleryRetryTestCase.test_celery_retry test after upgrading Celery results in the following error

======================================================================
ERROR: test_celery_retry (CeleryRetryTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "redacted/test.py", line 174, in some_task
    some_function(random_number)
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1123, in _mock_call
    raise result
SomeException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/redacted/test.py", line 2816, in test_celery_retry
    some_task.delay()
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay
    return self.apply_async(args, kwargs)
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 572, in apply_async
    return self.apply(args, kwargs, task_id=task_id or uuid(),
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 794, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 464, in trace_task
    I, R, state, retval = on_error(
  File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/redacted/test.py", line 177, in some_task
    raise self.retry(countdown=retry_countdown, max_retries=5)
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 691, in retry
    request = self.request
celery.exceptions.Retry: Retry in 1s

----------------------------------------------------------------------
Ran 1 test in 1.477s

FAILED (errors=1)

After several Google searches and failed attempts (e.g. calling apply() instead of delay(), setting CELERY_ALWAYS_EAGER to True, using the new default_retry_delay & autoretry_for attributes) I narrowed down the issue to this if block within Celerys retry method. Specifically, the removal of S.apply().get() within the if block which was introduced in this PR.

I was able to get my tests passing successfully after altering Celerys retry method to include S.apply() within this if block e.g.

        ret = Retry(exc=exc, when=eta or countdown, is_eager=is_eager, sig=S)

        if is_eager:
            # if task was executed eagerly using apply(),
            # then the retry must also be executed eagerly in apply method
            S.apply()
            if throw:
                raise ret
            return ret

@mchataigner could you help me understand why the removal of S.apply().get() was made? I'm trying to understand if this is a bug or if I'm missing something in my Celery configuration/task definitions.

auvipy · 2023-07-08T03:29:30Z

I'm running into, what I believe to be, a similar issue. I have a Django v3.2.19 application currently running Celery v4.3.0 and am attempting to upgrade to v5.2.7. After upgrading, a few test cases around task retry logic started failing which I believe to be related to this PR. Below is an example showcasing the issue

class SomeException(Exception):
    """ Custom Exception Class """
    pass


def some_function(number):
    if number < 0:
        raise SomeException
    return True


@app.task(bind=True)
def some_task(self):
    try:
        random_number = random.randint(0, 100)
        some_function(random_number)
    except SomeException as e:
        retry_countdown = int(random.uniform(2, 4) ** self.request.retries)
        raise self.retry(countdown=retry_countdown, max_retries=5)


class CeleryRetryTestCase(TestCase):

    @mock.patch('some_function')
    def test_celery_retry(self, mocked_some_function):
        retryable_exceptions = [
            SomeException(),
        ]
        mocked_some_function.side_effect = retryable_exceptions * 10
        with self.assertRaises(MaxRetriesExceededError):
            some_task.delay()

        # we should have called some_function() 6 times
        self.assertEqual(mocked_some_function.call_count, 6)

Running the CeleryRetryTestCase.test_celery_retry test after upgrading Celery results in the following error

======================================================================
ERROR: test_celery_retry (CeleryRetryTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "redacted/test.py", line 174, in some_task
    some_function(random_number)
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1123, in _mock_call
    raise result
SomeException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/redacted/test.py", line 2816, in test_celery_retry
    some_task.delay()
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay
    return self.apply_async(args, kwargs)
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 572, in apply_async
    return self.apply(args, kwargs, task_id=task_id or uuid(),
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 794, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 464, in trace_task
    I, R, state, retval = on_error(
  File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/redacted/test.py", line 177, in some_task
    raise self.retry(countdown=retry_countdown, max_retries=5)
  File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 691, in retry
    request = self.request
celery.exceptions.Retry: Retry in 1s

----------------------------------------------------------------------
Ran 1 test in 1.477s

FAILED (errors=1)

After several Google searches and failed attempts (e.g. calling apply() instead of delay(), setting CELERY_ALWAYS_EAGER to True, using the new default_retry_delay & autoretry_for attributes) I narrowed down the issue to this if block within Celerys retry method. Specifically, the removal of S.apply().get() within the if block which was introduced in this PR.

I was able to get my tests passing successfully after altering Celerys retry method to include S.apply() within this if block e.g.

        ret = Retry(exc=exc, when=eta or countdown, is_eager=is_eager, sig=S)

        if is_eager:
            # if task was executed eagerly using apply(),
            # then the retry must also be executed eagerly in apply method
            S.apply()
            if throw:
                raise ret
            return ret

@mchataigner could you help me understand why the removal of S.apply().get() was made? I'm trying to understand if this is a bug or if I'm missing something in my Celery configuration/task definitions.

thanks for tracing down to it. I'm not fully sure now if it was a regression! I would be happy to review a PR with that part re added and possibly more unit and integration tests

andrewdieken · 2023-07-08T04:20:55Z

@auvipy that would be great! I can get a PR opened early next week for your review

auvipy · 2023-07-08T04:48:07Z

just ping me after opening

Emilianissimo added the Issue Type: Bug Report label Feb 6, 2023

Emilianissimo changed the title ~~Django 2.2 + Celery 5.2.7 unit testing do not Retrying the task~~ Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task Feb 6, 2023

andrewdieken linked a pull request Jul 10, 2023 that will close this issue

Apply task upon eager retry #8367

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056

Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056

Emilianissimo commented Feb 6, 2023

open-collective-bot bot commented Feb 6, 2023

andrewdieken commented Jul 7, 2023

auvipy commented Jul 8, 2023

andrewdieken commented Jul 8, 2023

auvipy commented Jul 8, 2023

Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056

Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056

Comments

Emilianissimo commented Feb 6, 2023

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

open-collective-bot bot commented Feb 6, 2023

andrewdieken commented Jul 7, 2023

auvipy commented Jul 8, 2023

andrewdieken commented Jul 8, 2023

auvipy commented Jul 8, 2023