Add support for converting pyproject.toml-based Python3 packages. #1982

amdei · 2023-01-03T17:30:48Z

TL;DR:
It works! (With some limitations at the moment though, as this is work in progress currently)

git clone ....
cd fpm
python3 -m pip install importlib_metadata
ruby bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-install-lib=/usr/local/lib/python3.9/dist-packages/ --python-package-name-prefix python3 --python-bin=python3  typing_extensions

Limitations:

Does not support python2 in any way (obviously)

Temporary limitations (TODOs):

Do not support automatic dependencies extraction to the moment
Do not support license extraction to the moment
Checked only on Debian 11 amd64 (arm64 to go) so far
Not sure about applicability to 'native' packages
A lot of @todo's in code
Code quality is poor
Code-style is poor

I had no previous experience with ruby, so any suggestions/comments/guidelines are welcome.

@ObjatieGroba - could you please check Python part of code, and give your suggestions on how to improve it?

Fixes #1780
Fixes #1873
Fixes #1860
Fixes magma/magma#13847

amdei · 2023-01-06T11:16:41Z

After some reflections I've had a second thought.

Standard importlib.metadata is better that backported importlib_metadata
importlib.metadata is still not good enough to obtain all required metadata of not-yet-installed python package
I don't want to build wheel out of downloaded python package
But seems there is no solution without building wheel out of python package prior to converting it to some other format
It is possible and it is easy to get ALL required metadata via wheel API from pkginfo package

Going to revise my solution accordingly.
Stay tuned.

amdei · 2023-02-12T06:48:16Z

It seems to be working a little bit better now.
At least in terms of metadata retrieval.

Things to do:

1. Apply proper arch attribute to generated deb-package. At the moment all toml-based packages has Architecture: all;
2. Instruct wheel to not download dependencies for build package wheel;
3. For some packages fpm now silently dies with zero exit code. Need to get it work again;
4. Deal with packages dependencies. Not easy at the first glance.
5. Speed-up execution on processing toml packages. Now it 3 times slower that setup.py-based;
6. Perform code de-duplication. As now it contains significant amount of redundant fragments.
7. Cleanup code.
8. File some issues to pip, wheel, importlib, etc.
9. Obey --[no-]python-obey-requirements-txt fpm's option

Any suggestions/comments/guidelines are still welcome.
As well as package samples, where things go wrong.

jordansissel · 2023-02-13T02:58:20Z

Thanks for your work on this! I’ll take a look as soon as I can :)

ObjatieGroba · 2023-02-13T14:35:23Z

lib/fpm/package/pyfpm_wheel/get_metadata_wheel.py

+        return deps
+
+    def get_home_url(self, project_urls):
+        res = dict([i.strip() for i in x.split(',')] for x in project_urls)


Theren is no reason to generate dict.
Use next(filter) instead of dict.

return next((x.split(',', maxsplits=1)[1].strip() for x in project_urls if x.lstrip().startswith('Home')), None)

It appears that different packages put home URL into different keys.
For example Home vs. Homepage.

So let's stick to a current approach, if there is no more efficient solutions...

amdei · 2023-02-14T04:38:51Z

Bottom line:

Please update python wheel to latest version before trying anything: python3 -m pip install -U wheel - there was a bug, affected fpm, fixed only very recently.
It is slow. Very slow. Significantly slower that setup.py-based packaging;
Wasn't able to made it with architecture - it looks like there is simply no way to get arch neither from *.whl file nor from sdist via any API. But it contains in *.whl. But I'm not brave enough to parse it in that way (yet).
There is no alternatives found for setup.py's --install-lib, --install-data, --install-scripts and build_scripts --executable. So all fpm's --python-install-* options will not work (and currently silently ignored).
Staging with pip is broken. For example see here: Unexpected --target behavior with/without --upgrade pypa/pip#8799 Or it is intended for something else.
Documentation for pip install is rather short: https://docs.python.org/3/installing/index.html#installing-index Significantly shorter than one would expect.

amdei · 2023-04-08T23:14:18Z

After a while, I hack the Wheel metadata to get information if package is pure or not.
https://peps.python.org/pep-0491/
As it seems to be impossible (at least yet) via pkginfo.Wheel API: https://bugs.launchpad.net/pkginfo/+bug/2015657

amdei · 2023-04-10T00:30:10Z

It reached "works for me" stage.
Let's collect feedback before starting to polish things..

ObjatieGroba · 2023-04-12T20:20:38Z

lib/fpm/package/pyfpm/get_metadata.py

@@ -83,14 +83,17 @@ def run(self):

        if self.load_requirements_txt:
            requirement = open(self.requirements_txt).readlines()
+#            print('REQ-SETUP-PY-REQ-TXT:', requirement, file=sys.stderr)


unnecessary lines

ObjatieGroba · 2023-04-12T20:23:21Z

lib/fpm/package/python.rb

-    if !File.exist?(setup_py)
-      logger.error("Could not find 'setup.py'", :path => setup_py)
-      raise "Unable to find python package; tried #{setup_py}"
+    # @todo Swap it! - TOML priority is for debugging only!


It is time to swap

ObjatieGroba · 2023-04-12T20:23:30Z

lib/fpm/package/python.rb

+      logger.debug("Do job with pyproject.toml")
+      wheel_path = build_py_wheel(setup_py)
+      load_package_info_wheel(setup_py, wheel_path)
+      # install_to_staging_toml(setup_py)


unnecessary comment

ObjatieGroba · 2023-04-12T20:24:45Z

lib/fpm/package/python.rb

+        "#{Shellwords.escape(toml_metadata_code)}"
+
+      # @todo FIXME!
+      #      if attributes[:python_obey_requirements_txt?]


Doesn't this break backward compatibility?

ObjatieGroba · 2023-04-12T20:25:30Z

lib/fpm/package/python.rb

+    ::Dir.chdir(project_dir) do
+      flags = [ "--root", staging_path ]
+
+      # if !attributes[:python_install_lib].nil?


Lot's of commented code. Is it necessary?

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

…and name'.

JordanStopford · 2024-01-25T15:10:23Z

Amy update on this? A lot of packages no longer ship setup.py files so if we need to use fpm then we're stuck on old versions

cwegener · 2024-04-09T01:02:22Z

I don't understand what this does? It still requires a legacy setup.py file to be generated by the python project that is supposed to be converted. However, pretty much all PEP-517 build backends disable the support for generating setup.py legacy files by default, so they will not be able to be used by fpm. (some backends don't even offer an option at all to generate legacy setup.py files)

And if you have a source python project that has the option to generate a legeacy setup.py enabled, then such a project can already be used as a source in fpm without this change.

I am a bit confused what this change is supposed to do.

What is the intention of this change?

amdei · 2024-04-09T16:21:59Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented.
But if there is pyproject.toml file - it will be used instead.

cwegener · 2024-04-09T21:27:51Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented. But if there is pyproject.toml file - it will be used instead.

Thanks for the clarification. I'll need to read through the patch again then. 😁

amdei · 2024-04-17T00:43:36Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

cwegener · 2024-04-17T08:11:38Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

That's excellent. pip wheel will actually just invoke the Python build module which takes care of performing the right build process (legacy setup.py vs new PEP-517 build backends)

I will need to test out your PR and report the results.

Specifically, I will need to test with this PEP-517 project ... https://github.com/openai/openai-python

gmabey · 2024-05-08T14:08:22Z

Just an enthusiastic vote for more eyeballs on this merge request ... I would love ❤️ to see the project catch up to this shift in the python community.

cwegener · 2024-05-22T03:20:35Z

I will need to test out your PR and report the results.

Quick feedback from my initial testing of this patch:

Testing with

openai package from PyPI (which uses hatchling as the PEP 517 build backend)
Testing on Ubuntu 22.04 with built-in Python3.10

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel (and to a lesser degree pip download ... which is a separate story).

I think the main concern so far is the new dependency on the pkginfo Python library. E.g. on Ubuntu 22.04, the version of python3-pkginfo is a bit older and does not yet have support for version 2.3 Metadata file formats.

But a lot of wheels will now write 2.3 Metadata file formats.

Therefore, injecting or vendoring an up-to-date version of pkginfo is important.

CommandMC mentioned this pull request Jan 10, 2023

Update GH actions & specify Python version as dependency derrod/legendary#537

Merged

amdei force-pushed the python2deb-great-again branch from 419d841 to 9e16f11 Compare February 12, 2023 03:54

ObjatieGroba reviewed Feb 13, 2023

View reviewed changes

bugfood mentioned this pull request Mar 13, 2023

PEP518 - setup.py is deprecated #1987

Open

amdei marked this pull request as ready for review April 10, 2023 00:29

ObjatieGroba reviewed Apr 12, 2023

View reviewed changes

amdei added 14 commits September 23, 2023 01:31

Add support for converting pyproject.toml-based Python3 packages.

8d88bbc

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

Works again, with metadata from wheel.

c1b7e17

Imporve metadata parsing.

3cf33e8

Increase TOML priority over setup.py - for debugging only!

f94ef4e

Do not build wheels for dependencies.

fa08e5f

Fix python packages install flags in case of toml.

b3a66ce

Attempl to deal with dependencies correctly.

3df693a

Add check for python Wheelness.

c0c0a8a

Drop check for importlib.metadata python package.

a1777e6

Geting things easier.

c66239b

Fix staging install with --ignore-installed for pip install.

79411b1

Obtain package architecture via hackily hack.

451bb62

Fix native packages detection.

7e4fcac

Fix 'ERROR: The --python option must be placed before the pip subcomm…

43fefae

…and name'.

amdei force-pushed the python2deb-great-again branch from a9a1434 to 43fefae Compare September 22, 2023 22:49

amdei mentioned this pull request Nov 19, 2023

could not find setup.py in installable package #2029

Open

Stealthii mentioned this pull request Apr 24, 2024

PEP517 Support #2056

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for converting pyproject.toml-based Python3 packages. #1982

Add support for converting pyproject.toml-based Python3 packages. #1982

amdei commented Jan 3, 2023 •

edited

amdei commented Jan 6, 2023 •

edited

amdei commented Feb 12, 2023 •

edited

jordansissel commented Feb 13, 2023

ObjatieGroba Feb 13, 2023 •

edited

amdei Apr 8, 2023

amdei commented Feb 14, 2023 •

edited

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 •

edited

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

Add support for converting pyproject.toml-based Python3 packages. #1982

Are you sure you want to change the base?

Add support for converting pyproject.toml-based Python3 packages. #1982

Conversation

amdei commented Jan 3, 2023 • edited

amdei commented Jan 6, 2023 • edited

amdei commented Feb 12, 2023 • edited

jordansissel commented Feb 13, 2023

ObjatieGroba Feb 13, 2023 • edited

Choose a reason for hiding this comment

amdei Apr 8, 2023

Choose a reason for hiding this comment

amdei commented Feb 14, 2023 • edited

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 • edited

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

amdei commented Jan 3, 2023 •

edited

amdei commented Jan 6, 2023 •

edited

amdei commented Feb 12, 2023 •

edited

ObjatieGroba Feb 13, 2023 •

edited

amdei commented Feb 14, 2023 •

edited

amdei commented Apr 17, 2024 •

edited