use PyPI upload workflow from OpenAstronomy

[zacharyburnett]

Resolves SCSB-108

closes #8054

This PR changes the PyPI upload process to use the OpenAstronomy workflow instead of the STScI one. The workflow in this PR is configured to test the build with each pull request push, and then publish upon an official release.

This will facilitate catching build issues early on (ideally within the PR) before the release date. The reusable workflow uses cibuildwheel to build wheels for all supported Pythons on Linux, macOS X64, and macOS ARM64 (we don't currently have automated testing for ARM64, but we should soon with the on-prem regtesting effort)

We can't use this workflow until we make a PyPI token (with write / upload permission to packages) for the STScI maintainer account. Currently, we use a username and password to PyPI; however, this method will be deprecated at some point in the future in favor of tokens.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f13f8f5) 75.94% compared to head (d523bb6) 75.94%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7912   +/-   ##
=======================================
  Coverage   75.94%   75.94%           
=======================================
  Files         460      460           
  Lines       37629    37629           
=======================================
  Hits        28577    28577           
  Misses       9052     9052           

Flag	Coverage Δ		*Carryforward flag
nightly	77.37% <ø> (ø)		Carriedforward from f13f8f5

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zacharyburnett]

The following error occurred while building:
https://github.com/spacetelescope/jwst/actions/runs/6262235017/job/17003963430#step:7:703

                  Optimized (vendor) Blas libraries are not found.
                  Falls back to netlib Blas library which has worse performance.
                  A better performance should be easily gained by switching
                  Blas library.
                if self._calc_info(blas):
              /tmp/pip-install-x3ogdb8y/numpy_88331d5ab39a42d0ade0340a4d4d111e/numpy/distutils/system_info.py:1914: UserWarning:
                  Blas (http://www.netlib.org/blas/) libraries not found.
                  Directories to search for the libraries can be specified in the
                  numpy/distutils/site.cfg file (section [blas]) or by setting
                  the BLAS environment variable.

[zacharyburnett]

specifying CPython in the build targets fixed the issue

[zacharyburnett]

the source distribution issue in the OpenAstronomy workflow is now resolved (see OpenAstronomy/github-actions-workflows#169); I see no other blockers for merging

[pllim]

Still no Windows support? Didn't someone fixed it being broken on Windows at some point? Is it broken again?

[pllim]

Also if this is a high traffic repo and building the wheels is resource intensive, should consider only building them when a special label is applied for pull requests. Probably also wise to auto-cancel duplicate builds. See astropy on how this is done in production.

[pllim]

I left what comments I can. Someone from your team should approve.

I think this fixes #8054 . Thanks!

[nden]

Is the dev job fixed in a different PR?
Changes look good. I'm OK with merging as is and fixing the dev tests in another PR if that's what @zacharyburnett prefers.

[pllim]

Re: devdeps -- You need pyerfa-dev, see:

• For downstream maintainers: astropy-dev + numpy-dev now also needs pyerfa-dev astropy/astropy#15638

But even with that, I don't think you can resolve the numpy ABI failure. You have a lot of dependencies that build their own C-extensions. Anything that builds their own need to be built with numpy 2.0 to work with numpy 2.0 (drizzle, spherical-geometry, scikit-image, etc). NumPy claims that stuff built with numpy 2.0 would be backward compatible with 1.x (up to a point).

And if you have your own C-extensions, you might also run into compatibility issues in your own build after you resolve the dependencies, so be on the lookout for that.

I know that scikit-image is still struggle to provide a dev wheel that is compatible with numpy 2.0, please follow:

• Build nightly wheels with nightly NumPy 2.0 scikit-image/scikit-image#7251

tl;dr -- You probably can merge this as-is and deal with devdeps later because it is going to be painful.

[zacharyburnett]

Ok, when the tests pass I will merge this as-is then, and work on #7910 and #6847

[nden]

Approving - given that dev builds are going to be handled in a different PR.