Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing distutils for 3.12 compatibility #2210

Merged
merged 4 commits into from
May 17, 2024
Merged

Conversation

Cyanosite
Copy link
Contributor

Distutils has been deprecated in 3.10 and removed in 3.12 (https://docs.python.org/3.10/whatsnew/3.10.html#distutils-deprecated)

For 3.12 compatibility (#2129) I'm proposing to remove distutils.

distutils was only used for its StrictVersion class, this can be replaced by packaging's Version class in every instance, as this library is already a dependency this requires no further packages to be added as dependency.

For the use of the Version class, see: https://packaging.pypa.io/en/latest/version.html

@TobyRoseman
Copy link
Collaborator

Thanks @Cyanosite for this pull request. The change looks good to me.

CI: https://gitlab.com/coremltools1/coremltools/-/pipelines/1288882002

coremltools/_deps/__init__.py Outdated Show resolved Hide resolved
@Cyanosite
Copy link
Contributor Author

Cyanosite commented May 13, 2024

  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm getting the test pipelines to run locally so you don't have to test my PR every time I submit changes.
I ran into the above issue while trying to run the tests.
https://stackoverflow.com/questions/77364550/attributeerror-module-pkgutil-has-no-attribute-impimporter-did-you-mean
pkgutil.ImpImporter was long deprecated and has been removed in 3.12, due to this I will be trying to do something about this error too. I will get back to you once I have a working solution.

Alternatively I could work on that in a separate PR and just test with 3.10 for now

@TobyRoseman
Copy link
Collaborator

The test pipeline isn't going to work with Python 3.12. We'll need to update all of our test dependencies to version that support Python 3.12.

I recommend you stick with 3.10 for this pull request.

@Cyanosite
Copy link
Contributor Author

rootdir: /Users/cyanosite/Development/coremltools, configfile: pytest.ini
plugins: flake8-1.0.7, cov-5.0.0, xdist-2.5.0, sugar-1.0.0, mock-3.8.2, timeout-2.3.1, forked-1.6.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 0 items                                                                                                                                    

================================================================== warnings summary ==================================================================
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
  <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

<frozen importlib._bootstrap>:671
  <frozen importlib._bootstrap>:671: ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------- generated xml file: /Users/cyanosite/Development/coremltools/build/py-test-report.xml --------------------------------

Results (2.24s):
ERROR: module or package not found: coremltools.test (missing __init__.py?)

I cannot figure out why the pipeline doesn't work locally, seems to be the same error as in the pipeline on gitlab.
Flake reports no errors, that runs fine.

@TobyRoseman
Copy link
Collaborator

@Cyanosite
Copy link
Contributor Author

I fixed the remaining importerrors reported in the pipeline, should be good now.
Managed to get the tests working locally, the test that failed on the pipeline now runs for me with only one error.

>           assert(_HAS_KMEANS1D)
E           AssertionError

../../envs/coremltools-py3.10/lib/python3.10/site-packages/coremltools/models/neural_network/quantization_utils.py:404: AssertionError

This might be something to do with dependencies on my side rather than the pull request. It might run fine on the pipeline.
However the optimizations tests use quite a lot of swap so this might not be something I can run locally often.

@TobyRoseman
Copy link
Collaborator

@TobyRoseman TobyRoseman merged commit 9883e8d into apple:main May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants