Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependencies fail to resolve correctly when installing via egg #1179

Closed
bbatliner opened this issue Feb 9, 2021 · 2 comments · Fixed by #1180
Closed

Dependencies fail to resolve correctly when installing via egg #1179

bbatliner opened this issue Feb 9, 2021 · 2 comments · Fixed by #1180
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@bbatliner
Copy link
Contributor

Environment details

  • OS type and version: CentOS 7 / 8
  • Python version: Python 3.8.5 and Python 3.8.3
  • pip version: pip 20.1.1 and pip 19.3.1, respectively with the Python versions
  • google-api-python-client version: 1.12.8 (latest)

Steps to reproduce

  1. Create this minimal setup.py:
from setuptools import setup, find_packages

setup(
  name='FooModule',
  version='0.0.1',
  author='Foo Bar',
  author_email='author@example.com',
  description='foo',
  packages=find_packages(),
  install_requires=['google-api-python-client'],
)
  1. Ensure wheel and setuptools are installed.
[root@f369e3b27dbf foo_module]# pip3 list installed
Package    Version
---------- -------
pip        19.3.1
setuptools 41.6.0
wheel      0.36.2
  1. Build FooModule as an egg: python3 setup.py bdist_egg

  2. Try to install the egg: python3 -m easy_install dist/FooModule-0.0.1-py3.8.egg OR python3 setup.py install. This should fail! The error is error: google-auth 2.0.0.dev0 is installed but google-auth<2.0dev,>=1.21.1 is required by {'google-api-core'} See the full output below.

Click to expand
[root@f369e3b27dbf foo_module]# python3 setup.py install
running install
running bdist_egg
running egg_info
writing FooModule.egg-info/PKG-INFO
writing dependency_links to FooModule.egg-info/dependency_links.txt
writing requirements to FooModule.egg-info/requires.txt
writing top-level names to FooModule.egg-info/top_level.txt
reading manifest file 'FooModule.egg-info/SOURCES.txt'
writing manifest file 'FooModule.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying FooModule.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying FooModule.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying FooModule.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying FooModule.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying FooModule.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/FooModule-0.0.1-py3.8.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing FooModule-0.0.1-py3.8.egg
Copying FooModule-0.0.1-py3.8.egg to /usr/local/lib/python3.8/site-packages
Adding FooModule 0.0.1 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/FooModule-0.0.1-py3.8.egg
Processing dependencies for FooModule==0.0.1
Searching for google-api-python-client
Reading https://pypi.org/simple/google-api-python-client/
Downloading https://files.pythonhosted.org/packages/83/fc/98045b8c5e0ba12929d423e9ff6b742951bb846707539b18f19b27c6ddc3/google_api_python_client-1.12.8-py2.py3-none-any.whl#sha256=3c4c4ca46b5c21196bec7ee93453443e477d82cbfa79234d1ce0645f81170eaf
Best match: google-api-python-client 1.12.8
Processing google_api_python_client-1.12.8-py2.py3-none-any.whl
Installing google_api_python_client-1.12.8-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/google_api_python_client-1.12.8-py3.8.egg/EGG-INFO/requires.txt
Adding google-api-python-client 1.12.8 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/google_api_python_client-1.12.8-py3.8.egg
Searching for uritemplate<4dev,>=3.0.0
Reading https://pypi.org/simple/uritemplate/
Downloading https://files.pythonhosted.org/packages/bf/0c/60d82c077998feb631608dca3cc1fe19ac074e772bf0c24cf409b977b815/uritemplate-3.0.1-py2.py3-none-any.whl#sha256=07620c3f3f8eed1f12600845892b0e036a2420acf513c53f7de0abd911a5894f
Best match: uritemplate 3.0.1
Processing uritemplate-3.0.1-py2.py3-none-any.whl
Installing uritemplate-3.0.1-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
Adding uritemplate 3.0.1 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/uritemplate-3.0.1-py3.8.egg
Searching for six<2dev,>=1.13.0
Reading https://pypi.org/simple/six/
Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl#sha256=8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced
Best match: six 1.15.0
Processing six-1.15.0-py2.py3-none-any.whl
Installing six-1.15.0-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
Adding six 1.15.0 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/six-1.15.0-py3.8.egg
Searching for httplib2<1dev,>=0.15.0
Reading https://pypi.org/simple/httplib2/
Downloading https://files.pythonhosted.org/packages/15/7e/51e5bd333c0afa1c7bdbf98eb3b0ccf5167e2b1ecc8b4d13e9cc29291f81/httplib2-0.19.0-py3-none-any.whl#sha256=749c32603f9bf16c1277f59531d502e8f1c2ca19901ae653b49c4ed698f0820e
Best match: httplib2 0.19.0
Processing httplib2-0.19.0-py3-none-any.whl
Installing httplib2-0.19.0-py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/httplib2-0.19.0-py3.8.egg/EGG-INFO/requires.txt
Adding httplib2 0.19.0 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/httplib2-0.19.0-py3.8.egg
Searching for google-auth>=1.16.0
Reading https://pypi.org/simple/google-auth/
Downloading https://files.pythonhosted.org/packages/68/c3/6851ad3e029ac11e4beb6470b9465f4affce345cba8ef7cbf7fbd647c89a/google_auth-2.0.0.dev0-py2.py3-none-any.whl#sha256=50eda7bbeaed36ad17a87a94a770b9ec97e9a64add701942a4cd4dcbe291f3df
Best match: google-auth 2.0.0.dev0
Processing google_auth-2.0.0.dev0-py2.py3-none-any.whl
Installing google_auth-2.0.0.dev0-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/google_auth-2.0.0.dev0-py3.8.egg/EGG-INFO/requires.txt
Adding google-auth 2.0.0.dev0 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/google_auth-2.0.0.dev0-py3.8.egg
Searching for google-auth-httplib2>=0.0.3
Reading https://pypi.org/simple/google-auth-httplib2/
Downloading https://files.pythonhosted.org/packages/bd/4e/992849016f8b0c27fb604aafd0a7a724db16128906197bd1245c6f18e6a1/google_auth_httplib2-0.0.4-py2.py3-none-any.whl#sha256=aeaff501738b289717fac1980db9711d77908a6c227f60e4aa1923410b43e2ee
Best match: google-auth-httplib2 0.0.4
Processing google_auth_httplib2-0.0.4-py2.py3-none-any.whl
Installing google_auth_httplib2-0.0.4-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/google_auth_httplib2-0.0.4-py3.8.egg/EGG-INFO/requires.txt
Adding google-auth-httplib2 0.0.4 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/google_auth_httplib2-0.0.4-py3.8.egg
Searching for google-api-core<2dev,>=1.21.0
Reading https://pypi.org/simple/google-api-core/
Downloading https://files.pythonhosted.org/packages/c8/3f/f7faa8a96408167c03d6fe07e2bee9b383ec18a7a1117d32912eb0c1b78f/google_api_core-1.26.0-py2.py3-none-any.whl#sha256=002e44c533299aecd9dd265d200f9eacd9957cddd2c72e2cd1cb5cea127e972d
Best match: google-api-core 1.26.0
Processing google_api_core-1.26.0-py2.py3-none-any.whl
Installing google_api_core-1.26.0-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/google_api_core-1.26.0-py3.8.egg/EGG-INFO/requires.txt
Adding google-api-core 1.26.0 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/google_api_core-1.26.0-py3.8.egg
Searching for pyparsing<3,>=2.4.2
Reading https://pypi.org/simple/pyparsing/
Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl#sha256=ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b
Best match: pyparsing 2.4.7
Processing pyparsing-2.4.7-py2.py3-none-any.whl
Installing pyparsing-2.4.7-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
Adding pyparsing 2.4.7 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/pyparsing-2.4.7-py3.8.egg
Searching for rsa<5,>=3.1.4
Reading https://pypi.org/simple/rsa/
Downloading https://files.pythonhosted.org/packages/bf/87/dc7a6ebf0afbc602548627fa48e9c1147fa187233bf71d4c51c76a2cfb27/rsa-4.7-py3-none-any.whl#sha256=a8774e55b59fd9fc893b0d05e9bfc6f47081f46ff5b46f39ccf24631b7be356b
Best match: rsa 4.7
Processing rsa-4.7-py3-none-any.whl
Installing rsa-4.7-py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/rsa-4.7-py3.8.egg/EGG-INFO/requires.txt
Adding rsa 4.7 to easy-install.pth file
Installing pyrsa-decrypt script to /usr/local/bin
Installing pyrsa-encrypt script to /usr/local/bin
Installing pyrsa-keygen script to /usr/local/bin
Installing pyrsa-priv2pub script to /usr/local/bin
Installing pyrsa-sign script to /usr/local/bin
Installing pyrsa-verify script to /usr/local/bin

Installed /usr/local/lib/python3.8/site-packages/rsa-4.7-py3.8.egg
Searching for pyasn1-modules>=0.2.1
Reading https://pypi.org/simple/pyasn1-modules/
Downloading https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl#sha256=a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74
Best match: pyasn1-modules 0.2.8
Processing pyasn1_modules-0.2.8-py2.py3-none-any.whl
Installing pyasn1_modules-0.2.8-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/pyasn1_modules-0.2.8-py3.8.egg/EGG-INFO/requires.txt
Adding pyasn1-modules 0.2.8 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/pyasn1_modules-0.2.8-py3.8.egg
Searching for cachetools<5.0,>=2.0.0
Reading https://pypi.org/simple/cachetools/
Downloading https://files.pythonhosted.org/packages/bb/72/8df2e0dc991f1a1d2c6869404e7622e8ee50d80bff357dbb57c3df70305b/cachetools-4.2.1-py3-none-any.whl#sha256=1d9d5f567be80f7c07d765e21b814326d78c61eb0c3a637dffc0e5d1796cb2e2
Best match: cachetools 4.2.1
Processing cachetools-4.2.1-py3-none-any.whl
Installing cachetools-4.2.1-py3-none-any.whl to /usr/local/lib/python3.8/site-packages
Adding cachetools 4.2.1 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/cachetools-4.2.1-py3.8.egg
Searching for requests<3.0.0dev,>=2.18.0
Reading https://pypi.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e
Best match: requests 2.25.1
Processing requests-2.25.1-py2.py3-none-any.whl
Installing requests-2.25.1-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/requests-2.25.1-py3.8.egg/EGG-INFO/requires.txt
Adding requests 2.25.1 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/requests-2.25.1-py3.8.egg
Searching for pytz
Reading https://pypi.org/simple/pytz/
Downloading https://files.pythonhosted.org/packages/70/94/784178ca5dd892a98f113cdd923372024dc04b8d40abe77ca76b5fb90ca6/pytz-2021.1-py2.py3-none-any.whl#sha256=eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798
Best match: pytz 2021.1
Processing pytz-2021.1-py2.py3-none-any.whl
Installing pytz-2021.1-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
Adding pytz 2021.1 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/pytz-2021.1-py3.8.egg
Searching for protobuf>=3.12.0
Reading https://pypi.org/simple/protobuf/
Downloading https://files.pythonhosted.org/packages/b7/d3/37c36b3f5b125ac0d5b9f10d51dbecca355e24c233fd8e447c9879338ec2/protobuf-4.0.0rc2-py2.py3-none-any.whl#sha256=5df64b064b1b15e03768295939cbb5595733a02e8945dc033e56c6a9667a225c
Best match: protobuf 4.0.0rc2
Processing protobuf-4.0.0rc2-py2.py3-none-any.whl
Installing protobuf-4.0.0rc2-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/protobuf-4.0.0rc2-py3.8.egg/EGG-INFO/requires.txt
Adding protobuf 4.0.0rc2 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/protobuf-4.0.0rc2-py3.8.egg
Searching for packaging>=14.3
Reading https://pypi.org/simple/packaging/
Downloading https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl#sha256=67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a
Best match: packaging 20.9
Processing packaging-20.9-py2.py3-none-any.whl
Installing packaging-20.9-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/packaging-20.9-py3.8.egg/EGG-INFO/requires.txt
Adding packaging 20.9 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/packaging-20.9-py3.8.egg
Searching for googleapis-common-protos<2.0dev,>=1.6.0
Reading https://pypi.org/simple/googleapis-common-protos/
Downloading https://files.pythonhosted.org/packages/b4/84/1285880ebf18f91fdba9d11a0385f6c4b48f5b20459f84d4ee7fd3415ebc/googleapis_common_protos-1.53.0.dev2-py2.py3-none-any.whl#sha256=ef07666b75aa9f02e5bf8ea1d9c852f19df1f8d9aa4fa16d32f7ac965b66734e
Best match: googleapis-common-protos 1.53.0.dev2
Processing googleapis_common_protos-1.53.0.dev2-py2.py3-none-any.whl
Installing googleapis_common_protos-1.53.0.dev2-py2.py3-none-any.whl to /usr/local/lib/python3.8/site-packages
writing requirements to /usr/local/lib/python3.8/site-packages/googleapis_common_protos-1.53.0.dev2-py3.8.egg/EGG-INFO/requires.txt
Adding googleapis-common-protos 1.53.0.dev2 to easy-install.pth file

Installed /usr/local/lib/python3.8/site-packages/googleapis_common_protos-1.53.0.dev2-py3.8.egg
error: google-auth 2.0.0.dev0 is installed but google-auth<2.0dev,>=1.21.1 is required by {'google-api-core'}
  1. If you're paranoid about dependencies being left behind, run pip3 freeze | xargs pip3 uninstall -y and verify only pip, wheel, and setuptools are installed before continuing.
  2. Build FooModule as a wheel: python3 setup.py bdist_wheel
  3. Try to install the wheel: pip3 install dist/FooModule-0.0.1-py3-none-any.whl This will resolve the dependencies fine and succeed!

What's going on?

This is my best guess. This package's dependencies are written as follows:

install_requires = [
"httplib2>=0.15.0,<1dev",
"google-auth>=1.16.0",
"google-auth-httplib2>=0.0.3",
"google-api-core>=1.21.0,<2dev",
"six>=1.13.0,<2dev",
"uritemplate>=3.0.0,<4dev",
]

Notably, google-api-core itself has a dependency on google-auth, "google-auth >= 1.21.1, < 2.0dev".

Naively, these dependencies specify incompatible ranges. If google-auth>=1.16.0 is installed first, then the resulting google-auth dependency in google-api-core will fail to be satisfied (google-auth 2.0.0.dev0 is installed but google-auth<2.0dev,>=1.21.1 is required by {'google-api-core'}).

However, it seems like pip and/or the wheel format are able to figure this out, and install a compatible dependency. easy_install and/or the egg format are unable to figure this out, and an error occurs.

Is this something that the google-api-python-client team is interested in fixing? I think the fix would be as simple as removing the dependency on google-auth, or upper-bounding the dependency range to match google-api-core's dependency range.

Workarounds in the meantime

  1. Add an extra dependency google-auth<2.0dev to FooModule. This is what we're doing today. It might rely on the ordering of install_requires, which isn't guaranteed to be the installation order.
  2. Build FooModule as a wheel. This is the more mature solution. We hope to do this eventually.
@busunkim96
Copy link
Contributor

Hi @bbatliner,

Is this something that the google-api-python-client team is interested in fixing? I think the fix would be as simple as removing the dependency on google-auth, or upper-bounding the dependency range to match google-api-core's dependency range.

I didn't realize there wasn't an upper bound on google-auth on this library, and I would be fine with adding it for consistency.

I would still recommend moving to a wheel though, since both easy_install and bdist_egg are deprecated. https://packaging.python.org/guides/distributing-packages-using-setuptools/#packaging-your-project walks through the current recommendations for packaging projects. Is creating/using a source dist python3 setup.py sdist something that's feasible for your project if you're not able to build a wheel?

@bbatliner
Copy link
Contributor Author

Thanks for the prompt reply! I opened #1180 to bound the dependency range.

I concur moving to wheel is best and we're aware of the deprecation. Thanks for the info.

@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Feb 10, 2021
@busunkim96 busunkim96 added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed triage me I really want to be triaged. labels Feb 10, 2021
gcf-merge-on-green bot pushed a commit that referenced this issue Feb 28, 2021
For consistency with google-api-core, which specifies
`google-auth >= 1.21.1, < 2.0dev` as a dependency.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/google-api-python-client/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Fixes #1179 🦕
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants