Celery doesn't fetch settings properly

[lcd1232]

Checklist

• Celery 4.1.0
• python 3.6
• I have verified that the issue exists against the master branch of Celery.

Steps to reproduce

Install project - https://github.com/lcd1232/celery_django_test

$ git clone https://github.com/lcd1232/celery_django_test.git
$ cd celery_django_test
$ virtualenv -p python3 .env
$ source .env/bin/activate
$ pip install -r requirements.txt
$ ./manage.py runserver

The problem is that celery doesn't get settings CELERY_TASK_ROUTES https://github.com/lcd1232/celery_django_test/blob/752d45f43ba0a141ca23a36bb3a3ef76e9886391/celery_test/settings.py#L123 but if uncomment celery_test/celery.py:21-25 all works as expected.

Update

I localized the problem. https://github.com/celery/celery/blob/master/celery/app/utils.py#L217
For example if we have in settings.py TASK_ROUTES, is_in_new and is_in_old will be empty set because 'TASK_ROUTES' != 'task_routes'.
If we have in settings.py 'task_routes', variable source will not contain task_routes

[pachewise]

The actual old value is CELERY_ROUTES, but it still isn't honored by Celery for some reason. Doing some debugging, it looks like the settings value isn't really parsed into app.conf.task_routes (for some reason, the settings conf is added to the defaults).

[lcd1232]

@pachewise do you know how to fix this issue?

[pachewise]

not yet @lcd1232 - still trying to understand how detect_settings works. Maybe digging into the commit history will give us a better idea of what happened.

[georgepsarakis]

@pachewise @lcd1232 I do not see the issue exactly. If I do the following:

# settings.py
MYCELERY_TASK_ROUTES = {
    'celery_test.celery.*': {
        'queue': 'site.tasks.supertest',
    }
}

# Celery module
app = Celery('7vpn')
app.config_from_object('settings', namespace='MYCELERY')

The configuration contains the task_routes dictionary as expected. Could you please elaborate on what should be happening?

[pachewise]

@georgepsarakis turns out there were some misconfigurations, though there is a weird quirk that print(app.conf) shows 'task_routes': None, while print(app.conf['task_routes']) shows the defined setting (probably due to how the Settings object works).

@lcd1232 You forgot to update your settings.py to point to where your task is...

CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672' # change to match your AMQP
CELERY_TASK_ROUTES = {
    'main.tasks.*': {
        'queue': 'site.tasks.supertest',
    }
}

[georgepsarakis]

Perhaps we need to add the configuration example to the documentation?

[syllogismos]

I'm facing the same issue, what is the work around for this, I am not able to use CELERY_TASK_ROUTES to define custom queues based on tasks. I also found that app.conf['task_routes'] has the config I added, but not the app.conf object.

Same issue found in stackoverflow, I ended up defining the queue in the decorator itself like they suggested. Its working now, but I want to be able to have all the queues and tasks a the same location.

Django: 1.11
celery: 4.1.0
python: 3.5

[pachewise]

@syllogismos - there wasn't an issue for OP's particular case, they had misconfigured. what's your CELERY_TASK_ROUTES and celery app look like?

[syllogismos]

Mine is a django app, and I was trying to use different queues for different tasks. And SQS backed.. I tried both SQS and redis but in both cases, the messages are only going to the default queue.

So my CELERY_TASK_ROUTES in settings.py looked something like this.

CELERY_TASK_ROUTES = {'app.tasks.my_task': 'my_queue'}

I tried various config options like defining CELERY_TASK_QUEUES with exchanges, routing keys and etc. From the celery documentation. But nothing worked. I finally ended up just defining the queue at the task definition instead as stated in this stackoverflow answer/discussion.
https://stackoverflow.com/questions/46373866/celery-multiple-queues-not-working-properly-all-the-tasks-are-sent-to-default-q

[pachewise]

do you have an app/tasks.py module?

CELERY_TASK_ROUTES = {
    'app.tasks.my_task': { 
        'queue': 'my_queue'
    }
}

this should be in your settings.py, and then you'll have to load the settings from your app; for example:

app = Celery('7vpn') 
app.config_from_object('absolute.import.to.settings') # TODO replace with correct import path
# you should also be able to pass in the django.conf.settings object, but I haven't tried that

[syllogismos]

I did try that. It didn't work for me. But haven't tried the app.config_from_object('absolute.import.to.django.settings') I used the one from the documentation where they suggest to use django.conf.settings

Anyway I printed the app.conf object after loading the config, and I do see the routes settings when i print them using app.conf['task_routes'] but dont see them when I print the app.conf object. Also note that other celery settings are being applied, like celeyd_log_format and things like that. I only have problem with task_routes.

edit: I tried the absolute import to of the settings file in config_from_object it also uses the default queue even if I define something else. I even have the 'queue' keyword in the settings like you suggested. I even tried defining celery_task_queues with exchange key, routing key even thought in documentation its said that sqs and redis don't need this. And then used these queues in the routes config. None of it worked.

[blablacio]

It turns out that this get properly parsed and runs fine for me:

CELERY_QUEUES = (Broadcast('broadcast-tasks'),)
CELERY_ROUTES = {
    'app.tasks.my_task': {
        'queue': 'broadcast-tasks'
    }
}

CELERY_TASK_QUEUES and CELERY_TASK_ROUTES does not work and that's what I was trying initially.

I initialized the app with a custom namespace='CELERY' and I think that's where the issue comes from. I troubleshot a similar problem a few months ago and remember that removing the namespace helped.

Would be great to see the whole mess around old and new naming being fixed soon as it already bit quite some people. Not sure if it would be feasible to deprecate old naming scheme and only support the new one.

[arsenio]

I just went through the same thing (using Flask, not Django, for what it's worth). CELERY_TASK_ROUTES doesn't seem to work when using celery.conf.update(), but CELERY_ROUTES does.

[shitikanth]

Just wasted an hour on this. Guys, please fix the documentation. How hard can it be to rename some variables in the documentation?

[sparrowt]

If you come here confused about the difference between CELERY_ROUTES and CELERY_TASK_ROUTES (or CELERY_QUEUES & CELERY_TASK_QUEUES) then take a look at this comment on a related issue.

[evensonliu]

It may cuased by kombu.

at first my env were:
kombu-4.6.11
celery-4.4.7
celery could not get settings by config_from_object !

after changed to:
kombu-4.0.2
celery-4.0.2

problem solved!