threading.Thread.start() throws AttributeError while using gevent worker class on python-3.9.16 and Freebsd 11.4 release

[chethu2288]

• gevent version: gevent-21.8.0, greenlet-1.1.1 compiled and installed from source.
• Python version: Python 3.9.16 compiled and installed from source
• Operating System: FreeBSD 11.4-RELEASE

Description:

We have a Flask App which uses threading.Thread class to spawn threads for Async API requests.
Running this Flask App(Flask-1.1.2) using Gunicorn(gunicorn-20.0.4) and Gevent worker class throws error while starting threads. Need help to understand what is going wrong here or if I am missing something.

Stack trace from console when I invoke curl -k http://localhost:8000/count

thread.start()
 File “/var/python/lib/python3.9/threading.py”, line 899, in start
  _start_new_thread(self._bootstrap, ())
 File “/var/python/lib/python3.9/site-packages/gevent-21.8.0-py3.9-freebsd-11.4-RELEASE-amd64.egg/gevent/thread.py”, line 82, in start_new_thread
  greenlet = Greenlet.spawn(function, *args)
 File “src/gevent/greenlet.py”, line 687, in gevent._gevent_cgreenlet.Greenlet.spawn
AttributeError: type object ‘method’ has no attribute ‘start’

Below is the way I am running flask app using gunicorn by specifying gevent worker class:

/var/python/bin/python3 /var/python/lib/python3.9/site-packages/gunicorn-20.0.4-py3.9.egg/gunicorn/app/wsgiapp.py -w 1 -b 127.0.0.1:8000 -k gevent sampleapp:app

Code snip for sample flask app with which I can reproduce the issue.

from flask import Flask 
import threading

app = Flask(__name__)

@app.route('/') 
def hello():
    return 'Hello, World!'

@app.route('/count') 
def count():
    def do_work():
        for i in range(5):
            print(f'Processing item {i}')
            # Perform your work here

    # Create and start a new thread
    thread = threading.Thread(target=do_work)
    thread.start()

    return 'Work started in a separate thread!'

if __name__ == '__main__':
    app.run()

[jamadden]

• Obviously, I can't reproduce that on any tested platform with any version of gevent/greenlet.
• FreeBSD is not a tested platform. I would hope that everything works, but I can make no guarantees about that.
• That said, your version of greenlet and gevent are very old. Please try using the current versions. Likewise, your gunicorn version is even older; try updating that.
• It's very unusual to invoke gunicorn in that way, by providing a full path to a Python file inside the package. The usual way is with the gunicorn command that is provided by the gunicorn package, but python -m gunicorn.app.wsgiapp is also acceptable.
• This looks like some sort of monkey-patching problem. In some cases gunicorn's patching is known to be too late (and I don't think gunicorn is tested on FreeBSD either.) Does it work you monkey-patch yourself and use the Flask development server? i.e.,

from gevent import monkey; monkey.patch_all()
from flask import Flask
import threading 
app = Flask(__name__)
...
# Run me directly: python /path/to/app.py

Looking at the first frame in the traceback,

 File “/var/python/lib/python3.9/threading.py”, line 899, in start
  _start_new_thread(self._bootstrap, ())

What should a normally patched version of that look like?

>>> from gevent import monkey; monkey.patch_all()
True
>>> from threading import _start_new_thread
>>> _start_new_thread
<function start_new_thread at 0x101ee7d80>
>>> _start_new_thread.__module__
gevent.thread

gevent.thread.start_new_thread is very simple:

def start_new_thread(function, args=(), kwargs=None):
    if kwargs is not None:
        greenlet = Greenlet.spawn(function, *args, **kwargs)
    else:
        greenlet = Greenlet.spawn(function, *args) # <-- line 82
    return get_ident(greenlet)

It just calls a classmethod on the Greenlet class. That method is also very simple:

class Greenlet:
    @classmethod
    def spawn(cls, *args, **kwargs):
        g = cls(*args, **kwargs)
        g.start() # <-- line 687
        return g

So somehow, calling the Greenlet.spawn classmethod instantiates the class...but gets a method object in return? Obviously that makes no sense.

The only other advice I can offer you is to just debug through it. Set a breakpoint before calling thread.start and step through to see what's going wrong. I suspect that won't tell you much we haven't already seen, and that's because I'm suspicious about something being compiled wrong with the C accelerators (a mismatch between what it was compiled for and what it is running, or a compilation using a bad version of Cython, for example).

• If you're using gevent/greenlet provided by your system's package manager, try installing them from PyPI yourself.
• Try running in pure-Python mode (set the PURE_PYTHON environment variable) to eliminate most of the optional C accelerators.

$ PURE_PYTHON=1 python -m gunicorn.app.wsgiapp ...
# or 
$ PURE_PYTHON-1 gunicorn ...

Good luck!

[chethu2288]

@jamadden We had downloaded gevent source from pypi and it had .gitignore included under root folder. While adding this source to our code repo, we missed many .c and .html files because of .gitignore. Eventually compiling and packaging gevent from source that we maintained resulted in run time errors as reported by me in this thread. Note that there were no compile time errors because of missing .c files.
Thanks for your inputs in debugging this issue.