poetry never resolves package combination: google-cloud-aiplatform kfp #8475

kler · 2023-09-27T06:14:21Z

Poetry version: 1.6.1
Python version: 3.11.5
OS version and name: docker image python:3.11.5
pyproject.toml: Providing reproducible docker command instead
I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The following combination of packages never resolves: google-cloud-aiplatform kfp

Poetry is stuck in Resolving dependencies... after >10h.

Reproduced in both Native Macos 13.5.1 (22G90) and in Docker.

In Macos envs, cache and poetry.lock was removed like this, but despite this the error persists:

poetry env remove --all \
&& poetry cache clear --all -q . \
&& rm -rf $(poetry config cache-dir)/artifacts \
&& rm -f poetry.lock

Tested package combinations

❌ Doesn't resolve (within reasonable time)

google-cloud-aiplatform zenml mlflow gcsfs kfp <-- This combo was left running >10h
google-cloud-aiplatform kfp

✅ Does resolve

google-cloud-aiplatform zenml
google-cloud-aiplatform zenml mlflow
google-cloud-aiplatform zenml mlflow gcsfs
google-cloud-aiplatform gcsfs

How to reproduce

docker run --rm python:3.11.5 sh -c " \
curl -sSL https://install.python-poetry.org | python3 - --version=1.6.1 \
&& poetry --version \
&& mkdir app && cd app \
&& /root/.local/bin/poetry init --name=app --description=app --author=test@localhost --python='>=3.10,<3.12' -n \
&& /root/.local/bin/poetry add google-cloud-aiplatform kfp"

Output

Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.6.1)
Installing Poetry (1.6.1): Creating environment
Installing Poetry (1.6.1): Installing Poetry
Installing Poetry (1.6.1): Creating script
Installing Poetry (1.6.1): Done

Poetry (1.6.1) is installed now. Great!

To get started you need Poetry's bin directory (/root/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/root/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/root/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Creating virtualenv app-9TtSrW0h-py3.11 in /root/.cache/pypoetry/virtualenvs
Using version ^1.33.1 for google-cloud-aiplatform
Using version ^2.3.0 for kfp

Updating dependencies
Resolving dependencies...

The text was updated successfully, but these errors were encountered:

Davidnet · 2023-11-04T18:04:21Z

Still present in 1.7.0

Davidnet · 2023-11-29T19:06:38Z

Now at Nov29, and poetry 1.7.1 the packages:

$poetry add google-cloud-pipeline-components kfp

It fails with:

The following error occurred when trying to handle this error:


  OverrideNeeded

  {Package('google-api-core', '2.14.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-cloud-bigquery', '3.13.0'): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.4'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-api-core', '2.13.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.13.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.12.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.3'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.2'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}}

  at ~/.local/share/pypoetry/venv/lib/python3.11/site-packages/poetry/puzzle/provider.py:660 in complete_package
      656│                     current_overrides.update({dependency_package: package_overrides})
      657│                     overrides.append(current_overrides)
      658│ 
      659│             if overrides:
    → 660│                 raise OverrideNeeded(*overrides)
      661│ 
      662│         # Modifying dependencies as needed
      663│         clean_dependencies = []
      664│         for dep in dependencies:

The following error occurred when trying to handle this error:


  AssertionError

Davidnet · 2023-11-29T19:10:19Z

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <david.cardozo@me.com>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

HenriqueAJNB · 2024-02-16T13:33:02Z

I have the same problem:

Poetry version: 1.7.1
Python version: 3.10.12
OS version and name: Ubuntu 22.04.3 LTS
pyproject.toml

[tool.poetry]
name = "my-project"
version = "0.0.1"
description = ""
authors = []
readme = "README.md"
packages = [{ include = "my_pkg" }]

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
kfp = "1.8.22"
google-cloud-aiplatform = "1.25.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Issue

To reproduce the issue using the same environment, just run poetry install

It will fall into infinite dependency resolving.

$ poetry install
Creating virtualenv my-project in /home/.../.venv
Updating dependencies
Resolving dependencies... (2003.3s)

The logs using -vvv option shows what is the package that is causing the problem:

Found 0 compatible packages for grpcio-status
   1: fact: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: conflict: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.0rc1,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.51.0rc1)
   1: ! which is caused by "grpcio-status (1.51.0rc1) depends on protobuf (>=4.21.6)"
   1: ! thus: grpcio-status (>1.49.1,<1.50.0 || >1.50.0,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0) requires protobuf (>=4.21.6)
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.52.0rc1)
   1: ! which is caused by "grpcio-status (1.52.0rc1) depends on protobuf (>=4.21.6)"
...
   1: conflict: grpcio-status (1.56.2) depends on protobuf (>=4.21.6)
...
   1: conflict: grpcio-status (1.60.0) depends on protobuf (>=4.21.6)
...

Tons of those conflits appears, but no error or exception is raised.

The `pip` way

But, when I use built-in package venv instead of poetry it runs smootly.

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip3 install kfp==1.8.22 google-cloud-aiplatform==1.25.0

And the most important, pip can found a version for grpcio-status...

(.venv) $ pip3 show grpcio-status
Name: grpcio-status
Version: 1.48.2
Summary: Status proto mapping for gRPC
Home-page: https://grpc.io
Author: The gRPC Authors
Author-email: grpc-io@googlegroups.com
License: Apache License 2.0
Location: /home/.../.venv/lib/python3.10/site-packages
Requires: googleapis-common-protos, grpcio, protobuf
Required-by:

Attempts

I've tried to use the following attempts with no sucess:

Force grpcio-status = "1.48.2" in pyproject.toml
Be flexible with dependencies using kfp = "^1.8" and google-cloud-aiplatform = "^1"
Clear all caches by running poetry cache clear pypi --all and rm -rf ~/.cache/pypoetry/cache

Sum up

In short terms: this dependencies in pyproject.toml can't be installed using poetry, but they can using pip

Extra information

This dependencies could be installed normally in macOS Sonoma 14.1.1
The updated versions of kfp v2.x and google-cloud-platform it seems to work fine. The issue occurs in kfp v1.x only.

HenriqueAJNB · 2024-02-16T13:40:28Z

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <david.cardozo@me.com>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

@Davidnet, could you try to install both kfp and google-cloud-aiplatform now? Here it works, kfp released v2.7.0 recently, just want to be sure that issue is kfp v1.x related.

kler added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Sep 27, 2023

dimbleby linked a pull request Sep 30, 2023 that will close this issue

Fix for handling of version ranges python-poetry/poetry-core#645

Draft

dimbleby mentioned this issue Nov 3, 2023

Poetry enviroment that never resolves. #8617

Closed

4 tasks

dimbleby mentioned this issue Dec 8, 2023

Installing google-cloud-pipeline-components from PyPi hangs #8758

Closed

dimbleby mentioned this issue Dec 18, 2023

Version Strings Lexically Sorting so 2.2.0 < 2.2.0rc1 #8794

Closed

4 tasks

dimbleby mentioned this issue Feb 16, 2024

Infinite dependency resolving for kfp and google-cloud-aiplatform #8979

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

poetry never resolves package combination: google-cloud-aiplatform kfp #8475

poetry never resolves package combination: google-cloud-aiplatform kfp #8475

kler commented Sep 27, 2023

Davidnet commented Nov 4, 2023

Davidnet commented Nov 29, 2023

Davidnet commented Nov 29, 2023

HenriqueAJNB commented Feb 16, 2024 •

edited

HenriqueAJNB commented Feb 16, 2024

poetry never resolves package combination: google-cloud-aiplatform kfp #8475

poetry never resolves package combination: google-cloud-aiplatform kfp #8475

Comments

kler commented Sep 27, 2023

Issue

Tested package combinations

How to reproduce

Davidnet commented Nov 4, 2023

Davidnet commented Nov 29, 2023

Davidnet commented Nov 29, 2023

HenriqueAJNB commented Feb 16, 2024 • edited

Issue

The pip way

Attempts

Sum up

Extra information

HenriqueAJNB commented Feb 16, 2024

HenriqueAJNB commented Feb 16, 2024 •

edited

The `pip` way