-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't start a gevent based worker if httpx is intalled #9032
Comments
This issue was due to the latest openai python package using httpx .. I pinned it to the old version that didn't require it as my worker will run in a gevent pool. Then I installed some more AI libraries and they must have httpx as a dependancy and my gevent worker died again. My application does not use httpx so I assume some dependency is using it if available but it is failing under gevent causing the application to fail. If I just remove the httpx package everything works. However, there are no logs of failure anywhere. Nothing in my celery log file (with debug) nothing in my systemd service logs. My systemd server just continually restarts. How do I debug something like this? I would like to move my OpenAI calls to use an async worker pool as well but not sure where the problem is. |
Issue tracked to Trio (a dependency of httpx) not working under gevent (or eventlet). |
@pg1671 My workers are starting but httpx streams are sometimes broken, which verstion of OpenAI did you go with in the end? |
@Roman-Ka .. I don't use httpx yet (plan to). I just pinned to openai < 1.0.0 on the installation that was gevent based as it didn't use openai .. we are using prefork queues for that though will move to gevent workers when these issues are fixed. My gevent workers will just fail without any errors if httpx is installed which was ultimately the importing of trio (see above). The trio team are pretty proactive so hopefully they can resolve it but looks like they might need explicit support for gevent\eventlet etc in their package to resolve. I now just uninstall trio and am free to use httpx but not sure how well it will work under gevent. |
@Roman-Ka .. are you workers gevent or eventlet based? My prefork queues are ok with openai and I have the latest openai installed there. |
@pg1671 our workers are gevent based and we are using OpenAI streaming calls in them. They seem to running fine for the most part but sometimes the streams break and I was wondering if it's something to do with httpx not playing well with some celery component. We had other problems with Celery when it was running sometimes running things inside the event loop and sometimes outside. The actual error comes form inside OpenAI SDK and says smth like "sorry, try later", I am just hoping to find there is something on our end we can improve to prevent that. |
I think your error is just the openai servers being overloaded. Not much you can do other than try again. Are you using openAI > 1.0.0? OpenAI only uses httpx from 1.0.0. Do you have trio installed as well? |
If the python module httpx is installed then I cannot start a gevent based worker.
Checked against latest celery 5.4.0 running on Python 3.11 on ubuntu 20.04.
If I change the pool to prefork it starts.
If I uninstall httpx it works again with gevent.
I see nothing logged to my celery log file even with DEBUG level logging.
The command line output seems to indicate the worker is started ok.
celery multi v5.4.0 (opalescent)
but there are no processes for the worker if I run:
ps -aux | grep celery
No PID file is created.
There seems to be some conflict with celery\gevent and httpx.
The text was updated successfully, but these errors were encountered: