Direct URL PEP 508 support and installing sub-dependencies from Git

[dmfigol]

• Pip version: 10.0.1
• Python version: 3.7.0
• Operating system: macOS

I've spent the whole day reading about installing sub-dependencies from git and I am still struggling to find the correct solution.

My use-case

I forked an existing Python library (available on PyPi), added some changes that are unlikely to be merged upstream.
Some of my projects depend on the library fork. Previously it was an application, and I was using Pipenv to manage application dependencies, where I could easily specify to install specific branch/commit from Git in Pipfile. My application grew and now I am converting it to library (it is not going to be published on PyPi). I need to solve the problem of sub-dependencies from Git for the library.
I've read at least the following issues: #3610, #4187, #2124, #5384 and many others and of course PEP. When I finished reading I was even more confused than when I started.

From what I understood with implementation of PEP-508 in pip 10, I should be able to use:

setup(
    name='dmfigol',
...
    install_requires=[
        "requests",
        'smartsheet-python-sdk @ git+ssh://git@github.com/dmfigol/smartsheet-python-sdk@dev#egg=smartsheet-python-sdk-1.3.3',
    ],
)

But it does not work:

-> % pip install git+https://<private-git>/dmfigol/my-test-project.git
Collecting git+https://<private-git>/dmfigol/my-test-project.git
  ...
Direct url requirement (like smartsheet-python-sdk@ git+ssh://git@github.com/dmfigol/smartsheet-python-sdk@dev#egg=smartsheet-python-sdk-1.3.3) are not allowed for dependencies

My questions:

1. Did I do something wrong or this is not supported?
2. If not supported, is it going to be supported?
3. Do I understand correctly that I can also use similar syntax in pyproject.toml if I want to migrate from setup.py to something more declarative?
4. Is the only available solution today to use --process-dependency-links and dependency_links section in setup.py?
5. What happens when the upstream updates the library on PyPi to 1.3.4 or higher, and I would still like to use my forked version 1.3.3?

Thank you

UPD: currently working solution until #4187 is implemented if you stumble upon this thread:

setup(
    ...
    install_requires=[
        "smartsheet-python-sdk==10.1.3.3",
    ],
    dependency_links=[
        'https://github.com/dmfigol/smartsheet-python-sdk/archive/no-setuptools-scm.zip#egg=smartsheet-python-sdk-10.1.3.3'
    ],
...

[pradyunsg]

Did I do something wrong or this is not supported?

PEP 508 URL requirements as dependencies are not supported.

If not supported, is it going to be supported?

Yes. See below.

Do I understand correctly that I can also use similar syntax in pyproject.toml if I want to migrate from setup.py to something more declarative?

pyproject.toml only serves as a place to specify build time dependencies and to have configuration for tools. Anything other than that is implemented by the tools you're using.

Is the only available solution today to use --process-dependency-links and dependency_links section in setup.py?

Yes. That's the entire motivation behind #4187.

---

We disabled the ability to use PEP 508 URL requirements in dependencies, since we don't want that a package installed from PyPI to result in the pip reaching out to an arbitary web URL. Basically, pip install spam should not make pip reach out to anything except PyPI.

The current proposed solution in #4187 is to simply modify the blocking conditional so that it only blocks URL dependencies when a package is actually being installed from PyPI directly, so that any other mode of installation can use PEP 508 URL dependencies.

[dmfigol]

Thank you for detailed explanation, I really appreciate your help.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.