Status of Cython compatibility

[lysnikolaou]

First PR opened on cython/cython#6137.

This allows the test suite to run. All of the falures are related to the refnanny.

cython/cython#6147 enables vectorcall and fixes issues related to compiling in C99 mode.

[scoder]

cython/cython#6137 was merged

[rgommers]

Awesome, thanks @scoder and @lysnikolaou!

[ngoldbaum]

I noticed today there are some cython tests that assume PyObject has an ob_refcnt field. I also saw a segfault in parallel.parallel_exc_replace, which uses with gil. It doesn't particularly surprise me that with gil is broken.

We should probably have a brainstorming session about how to deal with prange code that does with gil.

[da-woods]

We should probably have a brainstorming session about how to deal with prange code that does with gil.

I think there's a number of levels of that:

• There's some user-code that assumes with gil can be used a lock around non-parallel bits of their code. That's never really been the right thing to do and probably isn't Cython's problem or your problem to fix.
• There's some general Cython-generated code that isn't thread safe. That isn't specifically a prange problem since you could likely reproduce bugs with any threading mechanism. It should probably be fixed for nogil builds but it'll likely be a long process of finding the corner-cases. Avoiding borrowed references may solve a lot.
• There's bit of Cython's internal code to do with prange/parallel that uses the GIL as a lock to juggle exception state between the Python interpreter and local variables. For that I suspect we'll need to guard it with a mutex in the nogil builds.
• Eventually we'll probably want a nice way of doing prange that contains Python code for nogil builds (without having to wrap everything in meaningless with gil: blocks). That's a future problem since it's a new feature rather than a bugfix.

[ngoldbaum]

Thanks for that @da-woods!

One thing in response to what you wrote:

Avoiding borrowed references may solve a lot.

This seems worth doing. Not every case is unsafe and it's probably unwise to manually convert every borrowed reference usage in human-authored C API code, but there's no reason for cython to use unsafe C API constructions when there are safe alternatives available.

I spent some time working on cython this morning and opened cython/cython#6154 to report what I found about the test suite in the nogil build.

[da-woods]

Avoiding borrowed references may solve a lot.

This seems worth doing. Not every case is unsafe and it's probably unwise to manually convert every borrowed reference usage in human-authored C API code, but there's no reason for cython to use unsafe C API constructions when there are safe alternatives available.

I meant setting the CYTHON_AVOID_BORROWED_REFS flag (which isn't perfect and doesn't cover everything but does a lot)