ENH: Support free-threaded python build (tracking issue)

[ngoldbaum]

Myself, @lysnikolaou, and @rgommers are working on a new project in collaboration with the Python Runtime team at Meta to add support for Python 3.13 nogil builds in the scientific python software stack.

Our initial minimal goal is a build of NumPy that runs and passes all the tests in a nogil build of Python.

Resources:

• PEP 703
• Fork of numpy 1.24 with early nogil support

Backlog

• Convert PyDict_GetItem and PyList_GetItem to use variants returning strong references (MNT: Replace C API usages that return borrowed references #26159)
  • Vendor pythoncapi-compat in a submodule (MNT: Adopt pythoncapi-compat? #26158)
• TSK: Make NumPy buildable on the nogil branch #26161
  • BUG: fix reference counting error in wrapping_method_resolve_descriptors #26190
  • TST: account for immortal objects in test_iter_refcount #26207
• Set up CPython 3.13 CI
  • 3.13 gil enabled build (TST: run the smoke tests on more python versions #26284)
  • --disable-gil build (TST: add basic free-threaded CI testing #26463)
• Make the test suite pass with the gil disabled
• Global state
  • Global caches need to be disabled, have locking, serialize creating read-only caches, or converted to thread-local caches.
    • Disabled:
      • MNT: disable the allocator cache for nogil builds #26251
      • MNT: disable the coercion cache for the nogil build #26294
      • NOGIL: Make loop data cache and dispatch cache thread-safe in nogil build #26348
    • Locking:
      • NOGIL: Make loop data cache and dispatch cache thread-safe in nogil build #26348
      • ENH: add locking around initializing the argparse cache #26430
    • Cache initialization moved to module initialization
      • MAINT: fix thread-unsafe cache initialization in PyUFunc_TrueDivisionTypeResolver #26497
    • Already thread safe:
      • npy_cache_import is implemented using thread-safe CPython C API constructs
    • Triaged low priority:
      • wrapping_array_method.c uses a freelist that is certainly not thread safe. It has no internal consumers besides _scaled_float_dtype to test the implementation so ignoring it for now.
  • Global settings and state like ErrState may need to be made per-thread or synchronized across threads.
    • Converted to thread-local state:
      • [NOGIL] thread local promotion state #26368

[ogrisel]

Maybe another interesting resource is the past work done by Sam Gross to patch numpy to make it work well enough with the original nogil fork of CPython 3.9: v1.24.0...colesbury:numpy:v1.24.0-nogil

Things might have drifted a bit since that time, but those patches are enough to get the scikit-learn test suite pass with nogil-CPython 3.9: see e.g. this run of the Linux_nogil pylatest_pip_nogil job that is scheduled run on a nightly basis on our Azure Pipelines CI.

[ngoldbaum]

It looks like numpy's main branch passes the full numpy test suite with the gil disabled. I've been testing using the nogil-integration branch in @colesbury's fork of CPython, but i tried just now using the CPython main branch and the test suite passes with the GIL disabled there as well.

There are still a ton of threading bugs I'm sure because of limited use of threads in the tests - we need better test coverage for threaded workflows. I'm already aware of a number of internal caches we'll likely need to disable or add locking for.

But, it's a good start!

[a-reich]

Just curious, are contributors planning to work on #24755 or is there any overlap where work for nogil would also help with allowing subinterpreters?

[ngoldbaum]

While this does have some overlap with work on subinterpreters since the C thread safety issues we're working on apply to subinterpreters as well, the bulk of the work required to support subinterpreters - converting numpy's extension types from single-phase to multi-phase initialization - is not required and we are not currently planning to do that.