Automatically set compiler flags to target PEP 384 Python limited API #26

lpsinger · 2021-02-09T15:41:27Z

PEP 384 introduced a Python limited API that allows you to build one wheel that targets all Python versions on a given platform.

Setuptools supports opting in to building wheels for the Python limited API by adding the following lines to setup.cfg:

[bdist_wheel]
py_limited_api = cp36  # replace with desired min Python version

However, there are two needed steps that Setuptools does not do automatically:

Pass the py_limited_api=True keyword argument to each Extension
Set the Py_LIMITED_API preprocessor macro to the version hex value for the desired minimum Python version

This patch teaches get_extension() to do those two missing steps automatically if the limited API is enabled in the setup.cfg file.

codecov · 2021-02-09T15:42:51Z

Codecov Report

Merging #26 (6547724) into main (51636f3) will decrease coverage by 1.39%.
Report is 94 commits behind head on main.
The diff coverage is 43.47%.

❗ Current head 6547724 differs from pull request most recent head cdfeeac. Consider uploading reports for the commit cdfeeac to get more accurate results

@@            Coverage Diff             @@
##             main      #26      +/-   ##
==========================================
- Coverage   74.65%   73.27%   -1.39%     
==========================================
  Files           9        9              
  Lines         513      535      +22     
==========================================
+ Hits          383      392       +9     
- Misses        130      143      +13

Files Changed	Coverage Δ
extension_helpers/_setup_helpers.py	`61.98% <43.47%> (-4.69%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Cadair

Thanks for the PR @lpsinger this looks really interesting.

Cadair · 2021-02-10T10:04:30Z

extension_helpers/_setup_helpers.py

+            ext.py_limited_api = True
+            ext.define_macros.append(('Py_LIMITED_API', version_hex))
+    else:
+        log.warn(_PEP_384_WARNING)


I don't understand the limitations of the limited API really, but does numpy compile against it? I am just wondering if this warning is too keen, i.e. if most of our users (scientific python packages with numpy C dep) are likely to be able to make use of this?

Downstream packages that compile against and link with Numpy can use the limited API. I maintain one such package. Here are the wheels: https://pypi.org/project/ligo.skymap/#files

The major limitation is that Cython support for the limited API is very, very new: cython/cython#3223

I wonder if you could make astropy core compile with this set 🤔 that would certainly be a good indicator that most things could use it.

Thanks for the extra info.

So just to make sure I understand, people will see the warning if they don't opt-in to this? If so I wonder if we should perhaps not do that, or at least make it clear to people how to get rid of the warning if they don't want to opt in to the limited API?

extension_helpers/_setup_helpers.py

Cadair · 2021-02-10T10:20:37Z

docs/using.rst

+
+Python limited API
+------------------
+


In a similar vein to my other comment, it would be nice to explain when using this is appropriate and when it isn't as I expect most people don't even know this exists (I didn't) and why they would want to use it.

Yes, you're right! It's not a well-known feature, although it seems to be well supported by pip. Part of the aim of this PR is to raise awareness by making it easy for Astropy affiliated packages to opt in.

PEP 384 introduced a Python limited API that allows you to build one wheel that targets all Python versions on a given platform. Setuptools supports opting in to building wheels for the Python limited API by adding the following lines to ``setup.cfg``: [bdist_wheel] py_limited_api = cp36 # replace with desired min Python version However, there are two needed steps that Setuptools does not do automatically: 1. Pass the `py_limited_api=True` keyword argument to each `Extension` 2. Set the `Py_LIMITED_API` preprocessor macro to the version hex value for the desired minimum Python version This patch teaches `get_extension()` to do those two missing steps automatically if the limited API is enabled in the ``setup.cfg`` file.

Cadair · 2021-02-10T15:08:22Z

It would be good if we could work out some tests for this, the difficultly of writing tests and the benefit of having them are both high in this package :D

lpsinger · 2021-02-10T15:10:26Z

It would be good if we could work out some tests for this, the difficultly of writing tests and the benefit of having them are both high in this package :D

For sure. I just wanted to get some feedback on the idea before writing a test case.

Cadair · 2021-02-10T15:18:52Z

👍 makes sense. The idea seems very good, if we can reduce the wheel build time by 1/3 I am very onboard.

astrofrog · 2021-02-11T08:37:26Z

Yes agree this would be great!

astrofrog · 2021-03-10T14:53:00Z

@lpsinger - I don't think there are any objections, so feel free to try and add a test for this if possible - and let us know if it's not.

lpsinger · 2021-03-10T15:05:46Z

BTW, it looks like the Cython support has come a long way, but Cython 3.0 is still in alpha and has been in alpha for a very long time.

astrofrog · 2021-03-10T15:11:08Z

@lpsinger - thanks for the update! I have a number of packages that don't use Cython that this could be very useful for, so looking forward to using it :)

lpsinger · 2021-03-10T17:22:12Z

What do people think about depending on alpha versions of Cython?

Cadair · 2021-04-29T14:59:43Z

I certainly wouldn't want to require it, but having it opt-in to use the new stuff wouldn't be too bad.

lpsinger · 2021-04-29T15:01:00Z

Sorry that progress on this stalled. I'll try and get back to it soon.

astrofrog · 2022-02-17T23:13:47Z

There is a minor conflict that needs to be resolved - could you rebase?

I'm really excited about trying this out as once thing that frustrates me is to keep having to build wheels for newer Pythons for packages that are ultra-stable and haven't had a new release for a couple of years. One practical question I have (unrelated to extension-helpers) is how to properly test the resulting wheels - I'm assuming one would build wheels with e.g. cibuildwheel for say cp36* but cibuildwheel doesn't really have a mechanism for building on Python 3.6 and testing the wheel on Python 3.9. @lpsinger do you just build and test the 3.6 wheels and then trust it will work on more recent Python versions?

lpsinger · 2022-02-18T02:32:19Z

@lpsinger do you just build and test the 3.6 wheels and then trust it will work on more recent Python versions?

Yes. And you're right that cibuildwheel doesn't do this yet... pypa/cibuildwheel#78

astrofrog · 2022-02-18T14:36:32Z

I've experimented with the limited API a little and I have an idea for a test to add, which is to add an extension that actually does not work with the limited API, and then try and build and use the package and check that it gives an error about not being able to load a symbol (this will tell us that the limited API option is indeed being passed through). Checking a working example is interesting too but not sufficient to really check things are working since it would work fine without the limited API option.

lpsinger · 2022-02-18T15:36:17Z

@lpsinger do you just build and test the 3.6 wheels and then trust it will work on more recent Python versions?

I'm sorry, I misspoke. I haven't tried this with cibuildwheel yet. For my own projects that use the limited API, I build once using Python 3.6 (or 3.7) and then install the wheels and run under all supported versions of Python.

astrofrog · 2022-02-18T23:45:45Z

So I just realized that the main code changes here depend on get_distutils_option which has since been removed since distutils is deprecated and direct use of setup.py is also deprecated. Just to check, do we actually need any changes in extension-helpers since users really just need to edit setup.cfg and then add the required option in Extension? I guess there is one case (single Cython files) where we auto-generate the Extension, and in that case we might want to add the correct flag if the limited API is set in setup.cfg?

lpsinger · 2022-02-23T02:54:45Z

Just to check, do we actually need any changes in extension-helpers since users really just need to edit setup.cfg and then add the required option in Extension?

Normally you have two make two changes to your setup.cfg:

Add py_limited_api = cp36 to the [bdist_wheel] section
Add the preprocessor definition to each of your extensions

The purpose of this PR is to make it so that the developer only has to do step 1, and step 2 is done automatically.

lpsinger · 2022-02-23T03:09:00Z

So I just realized that the main code changes here depend on get_distutils_option which has since been removed since distutils is deprecated and direct use of setup.py is also deprecated.

Setuptools does not yet support C extension modules in setup.cfg-only projects (pypa/setuptools#2220). (Cython .pyx extensions are supported, but that's a different matter.) Would it be a problem to bring back get_distutils_option?

astrofrog · 2022-02-23T08:32:16Z

Just a quick note that with #33 we can use extension helpers with setup.cfg only projects. I would prefer simply advocating that the option to Extension for the limited API is done explicitly in setup_package.py files rather than adding back distutils-related functionality.

Cadair · 2022-02-23T09:04:21Z

Isn't the use of get_distutils_option here just parsing the config file? We already do that in other ways?

I quite like the idea of including this in a nice easy way, it seems like a great feature. Not at the expense of relying on distutils again though having just removed it.

astrofrog · 2022-02-23T11:21:57Z

Ah yes my bad, we should just parse the config file directly to check for the option

astrofrog · 2022-02-23T14:02:28Z

@lpsinger - see here:

extension-helpers/extension_helpers/__init__.py

Lines 17 to 21 in 55b7859

    
           cfg = ConfigParser() 
        
           cfg.read(config_files[0]) 
        
           if (cfg.has_option("extension-helpers", "use_extension_helpers") and 
        
                   cfg.get("extension-helpers", "use_extension_helpers").lower() == 'true'): 
        
               distribution.ext_modules = get_extensions()

For an example of how we parse an option from setup.cfg

ConorMacBride · 2022-05-23T13:19:40Z

extension_helpers/_setup_helpers.py

+def _version_info_to_version_hex(major=0, minor=0, micro=0,
+                                 releaselevel=0, serial=0):
+    return f'0x{major:0>2}{minor:0>2}{micro:0>2}{releaselevel:0>2}{serial:0>2}'


Based on the Python API and ABI Versioning docs I think this would need to be:

Suggested change

def _version_info_to_version_hex(major=0, minor=0, micro=0,

releaselevel=0, serial=0):

return f'0x{major:0>2}{minor:0>2}{micro:0>2}{releaselevel:0>2}{serial:0>2}'

def _version_info_to_version_hex(major=0, minor=0):

"""Returns a PY_VERSION_HEX for {major}.{minor).0"""

return f'0x{major:02x}{minor:02x}0000'

Currently for cp310, {10:0>2} would give 10 instead of 0a~~, and the release level should be f for a final release~~. (Edit: according to this python doc setting the release level to 0 should be fine.)

extension_helpers/_setup_helpers.py

ConorMacBride · 2022-05-23T13:32:54Z

extension_helpers/_setup_helpers.py

+    abi = get_distutils_option('py_limited_api', ['bdist_wheel'])
+    if abi:
+        version_info = _abi_to_version_info(abi)
+        if version_info:


This isn't raising an error/warning if py_limited_api is specified but does not conform to the pattern, however, pypa/wheel will raise an error in this case so this should be fine.

extension_helpers/_setup_helpers.py

astrofrog · 2022-05-23T14:59:22Z

@lpsinger just to check, is this something you might have time to work on in the near future? If not, one of us could maybe try and wrap it up - just let me know!

lpsinger · 2022-05-23T15:03:55Z

@lpsinger just to check, is this something you might have time to work on in the near future? If not, one of us could maybe try and wrap it up - just let me know!

I think I can perhaps do it next week?

astrofrog · 2022-05-23T15:05:58Z

That would be great, thanks! :)

astrofrog · 2022-08-19T08:53:49Z

@lpsinger - just to check, is this something you would have time to work on?

lpsinger · 2022-08-19T12:15:54Z

Perhaps late next week.

astrofrog · 2022-08-24T12:22:41Z

Just a note that I started experimenting with Cython and the limited API and I can't get it to work even with a simple example:

https://groups.google.com/g/cython-users/c/-hasuL2VEdo/m/LFaf70PYAwAJ?utm_medium=email&utm_source=footer

This is even with Cython 3.0.0a11. Unless I'm missing something, with this PR we should probably add a note that things won't work for Cython code yet.

Co-authored-by: Conor MacBride <conor@macbride.me>

lpsinger · 2023-07-26T16:41:33Z

Cython 3.0.0 is out now and has initial support for Limited API](https://cython.readthedocs.io/en/latest/src/changes.html#initial-support-for-limited-api) (though nontrivial Cython extensions might fail). It looks like cibuildwheel supports it too.

Regardless of the Cython status, this is still valuable for projects like astropy-healpix that have C extensions but not through Cython.

I'm going to try to clean this PR up now.

Cadair reviewed Feb 10, 2021

View reviewed changes

lpsinger force-pushed the py-limited-api branch from 3b744c9 to 6547724 Compare February 10, 2021 14:03

astrofrog mentioned this pull request Mar 15, 2022

Release on PyPI from time to time? #38

Closed

Cadair mentioned this pull request May 23, 2022

Use Python Limited API sunpy/sunpy#6171

Merged

ConorMacBride reviewed May 23, 2022

View reviewed changes

Update extension_helpers/_setup_helpers.py

cdfeeac

Co-authored-by: Conor MacBride <conor@macbride.me>

Automatically set compiler flags to target PEP 384 Python limited API #26

Are you sure you want to change the base?

Automatically set compiler flags to target PEP 384 Python limited API #26

Conversation

lpsinger commented Feb 9, 2021

codecov bot commented Feb 9, 2021 • edited

Codecov Report

Cadair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cadair commented Feb 10, 2021 • edited

lpsinger commented Feb 10, 2021

Cadair commented Feb 10, 2021

astrofrog commented Feb 11, 2021

astrofrog commented Mar 10, 2021

lpsinger commented Mar 10, 2021

astrofrog commented Mar 10, 2021

lpsinger commented Mar 10, 2021

Cadair commented Apr 29, 2021

lpsinger commented Apr 29, 2021

astrofrog commented Feb 17, 2022

lpsinger commented Feb 18, 2022

astrofrog commented Feb 18, 2022

lpsinger commented Feb 18, 2022

astrofrog commented Feb 18, 2022

lpsinger commented Feb 23, 2022

lpsinger commented Feb 23, 2022

astrofrog commented Feb 23, 2022 • edited

Cadair commented Feb 23, 2022

astrofrog commented Feb 23, 2022

astrofrog commented Feb 23, 2022

ConorMacBride May 23, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astrofrog commented May 23, 2022

lpsinger commented May 23, 2022

astrofrog commented May 23, 2022

astrofrog commented Aug 19, 2022

lpsinger commented Aug 19, 2022

astrofrog commented Aug 24, 2022

lpsinger commented Jul 26, 2023

codecov bot commented Feb 9, 2021 •

edited

Cadair commented Feb 10, 2021 •

edited

astrofrog commented Feb 23, 2022 •

edited

ConorMacBride May 23, 2022 •

edited