First apply_async call to task of geventbased celery worker always fails

[karthikkj172]

I am running celery worker as a daemon in uwsgi:

attach-daemon2 = reloadsignal=1,stopsignal=3,control=1,cmd=/bin/celery -A <worker_module> worker -Q gen_queue --pool=gevent -Ofair --loglevel=info --concurrency=3 --heartbeat-interval 60 -n celery@%n

I have made sure to add monkey patching at both my celery app startup code and the task definition code.

When running a celery task on gevent pool:
task_id=custom_io_task.apply_async(args=("xyz"),task_id=task_id_gen())

I always get the following error at both the celery task log as well as the caller.
In my code , I have this call multiple times and I see trace only the first time.

[2023-11-28 05:55:25,628: WARNING/MainProcess] ImportError
[2023-11-28 05:55:25,628: WARNING/MainProcess] :
[2023-11-28 05:55:25,628: WARNING/MainProcess] libffi-0ff6ce5e.so.6.0.4: cannot open shared object file: No such file or directory
thread '<unnamed>' panicked at 'Python API call failed', /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/err/mod.rs:790:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2023-11-28 05:55:25,634: WARNING/MainProcess] Traceback (most recent call last):
[2023-11-28 05:55:25,634: WARNING/MainProcess]   File "/lib/python3.8/site-packages/celery/concurrency/base.py", line 29, in apply_target
    ret = target(*args, **kwargs)
[2023-11-28 05:55:25,634: WARNING/MainProcess]   File "/lib/python3.8/site-packages/celery/app/trace.py", line 649, in fast_trace_task
    R, I, T, Rstr = tasks[task].__trace__(
[2023-11-28 05:55:25,634: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/app/trace.py", line 441, in trace_task
    task.backend.store_result(
[2023-11-28 05:55:25,634: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/base.py", line 528, in store_result
    self._store_result(task_id, result, state, traceback,
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 47, in _inner
    return fun(*args, **kwargs)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 115, in _store_result
    session = self.ResultSession()
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 106, in ResultSession
    return session_manager.session_factory(
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 87, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 56, in create_session
    engine = self.get_engine(dburi, **kwargs)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 53, in get_engine
    return create_engine(dburi, poolclass=NullPool, **kwargs)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 423, in create_engine
    return strategy.create(*args, **kwargs)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/dialects/mysql/pymysql.py", line 62, in dbapi
    return __import__("pymysql")
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/__init__.py", line 59, in <module>
    from . import connections  # noqa: E402
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/connections.py", line 13, in <module>
    from . import _auth
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/_auth.py", line 9, in <module>
    from cryptography.hazmat.primitives import serialization, hashes
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/__init__.py", line 7, in <module>
    from cryptography.hazmat.primitives._serialization import (
[2023-11-28 05:55:25,635: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/_serialization.py", line 11, in <module>
    from cryptography.hazmat.primitives.hashes import HashAlgorithm
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/hashes.py", line 10, in <module>
    from cryptography.hazmat.bindings._rust import openssl as rust_openssl
[2023-11-28 05:55:25,636: WARNING/MainProcess] pyo3_runtime.PanicException: Python API call failed
[2023-11-28 05:55:25,636: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2023-11-28 05:55:25,636: WARNING/MainProcess] Traceback (most recent call last):
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/concurrency/base.py", line 38, in apply_target
    reraise(WorkerLostError, WorkerLostError(repr(exc)),
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/exceptions.py", line 108, in reraise
    raise value.with_traceback(tb)
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/concurrency/base.py", line 29, in apply_target
    ret = target(*args, **kwargs)
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/app/trace.py", line 649, in fast_trace_task
    R, I, T, Rstr = tasks[task].__trace__(
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/app/trace.py", line 441, in trace_task
    task.backend.store_result(
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/base.py", line 528, in store_result
    self._store_result(task_id, result, state, traceback,
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 47, in _inner
    return fun(*args, **kwargs)
[2023-11-28 05:55:25,636: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 115, in _store_result
    session = self.ResultSession()
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/__init__.py", line 106, in ResultSession
    return session_manager.session_factory(
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 87, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 56, in create_session
    engine = self.get_engine(dburi, **kwargs)
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/backends/database/session.py", line 53, in get_engine
    return create_engine(dburi, poolclass=NullPool, **kwargs)
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 423, in create_engine
    return strategy.create(*args, **kwargs)
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/sqlalchemy/dialects/mysql/pymysql.py", line 62, in dbapi
    return __import__("pymysql")
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/__init__.py", line 59, in <module>
    from . import connections  # noqa: E402
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/connections.py", line 13, in <module>
    from . import _auth
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/pymysql/_auth.py", line 9, in <module>
    from cryptography.hazmat.primitives import serialization, hashes
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/__init__.py", line 7, in <module>
    from cryptography.hazmat.primitives._serialization import (
[2023-11-28 05:55:25,637: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/_serialization.py", line 11, in <module>
    from cryptography.hazmat.primitives.hashes import HashAlgorithm
[2023-11-28 05:55:25,638: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/cryptography/hazmat/primitives/hashes.py", line 10, in <module>
    from cryptography.hazmat.bindings._rust import openssl as rust_openssl
[2023-11-28 05:55:25,638: WARNING/MainProcess] billiard.exceptions.WorkerLostError: PanicException('Python API call failed')
[2023-11-28 05:55:25,638: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2023-11-28 05:55:25,638: WARNING/MainProcess] Traceback (most recent call last):
[2023-11-28 05:55:25,638: WARNING/MainProcess]   File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
[2023-11-28 05:55:25,638: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/concurrency/base.py", line 41, in apply_target
    callback(ExceptionInfo())
[2023-11-28 05:55:25,638: WARNING/MainProcess]   File "  /lib/python3.8/site-packages/celery/worker/request.py", line 725, in on_success
    failed, retval, runtime = failed__retval__runtime
[2023-11-28 05:55:25,638: WARNING/MainProcess] TypeError: cannot unpack non-iterable ExceptionInfo object
[2023-11-28 05:55:25,638: WARNING/MainProcess] 2023-11-28T05:55:25Z
[2023-11-28 05:55:25,638: WARNING/MainProcess]
[2023-11-28 05:55:25,638: WARNING/MainProcess] <Greenlet at 0x7f041c680d00: apply_target(<function fast_trace_task at 0x7f04243e9280>, ('module_execution_lib.custom_io_distributed_task., {}, <bound method create_request_cls.<locals>.Request., <bound method Request.on_accepted of <Request: mod, timeout=None, timeout_callback=<bound method Request.on_timeout of <Request: modu)> failed with TypeError

could someone help me out here?