ImportError: cannot import name '_state' from partially initialized module 'hydra_manager.celery' (most likely due to a circular import)

[wolfred24]

Hi guys, im integrating Celery with django and im always getting the following error (i got like 3 days trying to solve this):

ImportError: cannot import name '_state' from partially initialized module 'hydra_manager.celery' (most likely due to a circular import)
This is my code at hydra_manager/init.py (this is the code causing the problem):

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
#
__all__ = ("celery_app",)

And this is my hydra_manager/celery.py file:

import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hydra_manager.settings')

app = Celery('hydra_manager')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.
app.autodiscover_tasks()


@app.task(bind=True, ignore_result=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

I have am following the instructions at the oficial docs.

Redis info:

6141:C 07 Nov 2023 16:30:42.102 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6141:C 07 Nov 2023 16:30:42.102 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=6141, just started
6141:C 07 Nov 2023 16:30:42.102 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
6141:M 07 Nov 2023 16:30:42.102 * Increased maximum number of open files to 10032 (it was originally set to 256).
6141:M 07 Nov 2023 16:30:42.102 * monotonic clock: POSIX clock_gettime
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 7.2.3 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                  
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 6141
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           https://redis.io       
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

6141:M 07 Nov 2023 16:30:42.103 # WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.
6141:M 07 Nov 2023 16:30:42.103 * Server initialized
6141:M 07 Nov 2023 16:30:42.103 * Ready to accept connections tcp
^C6141:signal-handler (1699396263) Received SIGINT scheduling shutdown...
6141:M 07 Nov 2023 16:31:03.608 * User requested shutdown...
6141:M 07 Nov 2023 16:31:03.608 * Saving the final RDB snapshot before exiting.
6141:M 07 Nov 2023 16:31:03.617 * DB saved on disk
6141:M 07 Nov 2023 16:31:03.617 # Redis is now ready to exit, bye bye...

Django info:

Name: Django
Version: 4.2.4
Summary: A high-level Python web framework that encourages rapid development and clean, pragmatic design.
Home-page: https://www.djangoproject.com/
Author: Django Software Foundation
Author-email: foundation@djangoproject.com
License: BSD-3-Clause
Location: /Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages

Full error log:

/Users/cesar/developement/Hydra-Manager/hydra_manager/__init__.py changed, reloading.
Traceback (most recent call last):
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 74, in execute
    super().execute(*args, **options)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 81, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
           ^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/cesar/developement/Hydra-Manager/hydra_manager/__init__.py", line 3, in <module>
    from .celery import app as celery_app
  File "/Users/cesar/developement/Hydra-Manager/hydra_manager/celery.py", line 3, in <module>
    from celery import Celery
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/celery/local.py", line 460, in __getattr__
    module = __import__(self._object_origins[name], None, None,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/celery/app/__init__.py", line 2, in <module>
    from hydra_manager.celery import _state
ImportError: cannot import name '_state' from partially initialized module 'hydra_manager.celery' (most likely due to a circular import) (/Users/cesar/developement/Hydra-Manager/hydra_manager/celery.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cesar/developement/Hydra-Manager/manage.py", line 22, in <module>
    main()
  File "/Users/cesar/developement/Hydra-Manager/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/core/management/base.py", line 425, in run_from_argv
    connections.close_all()
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/connection.py", line 84, in close_all
    for conn in self.all(initialized_only=True):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/connection.py", line 76, in all
    return [
           ^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
                ^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/cesar/developement/Hydra-Manager/hydra_manager/__init__.py", line 3, in <module>
    from .celery import app as celery_app
  File "/Users/cesar/developement/Hydra-Manager/hydra_manager/celery.py", line 3, in <module>
    from celery import Celery
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/celery/local.py", line 460, in __getattr__
    module = __import__(self._object_origins[name], None, None,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cesar/developement/Hydra-Manager/venv/lib/python3.11/site-packages/celery/app/__init__.py", line 2, in <module>
    from hydra_manager.celery import _state
ImportError: cannot import name '_state' from partially initialized module 'hydra_manager.celery' (most likely due to a circular import) (/Users/cesar/developement/Hydra-Manager/hydra_manager/celery.py)
D

This is my project structure:

[wolfred24]

This is what is causing the error:

[tanja-ovc]

@wolfred24 Hello, Cesar! I was hoping that maybe you managed to solve this problem. I have exactly the same one and absolutely no idea what to do... It's a pity no one answered :(