Error installing git+file VCS references

[sbidoul]

Environment

• pip version: 20.0
• Python version: 3.7
• OS: linux

Description

The following installation of a PEP 440 local VCS reference fails: (using bash to reproduce)

$ git clone https://github.com/pypa/packaging
...
$ pip install "packaging @ git+file://$PWD/packaging"
ERROR: Invalid requirement: 'packaging @ git+file:///home/.../packaging'
Hint: It looks like a path. File 'packaging @ git+file:///home/.../packaging' does not exist.

Expected behavior

It should work, as does the pip-specific URL form:

$ pip install "git+file://$PWD/packaging#egg=packaging"

and as does the PEP 440 form with a remote VCS URL:

$ pip install "packaging @ git+https://github.com/pypa/packaging"

[pradyunsg]

"packaging @ git+https://github.com/pypa/packaging"

This isn't a valid PEP 508 requirement (see https://www.python.org/dev/peps/pep-0508/#examples) -- it's a weird abomination of pip-specific VCS references and PEP 508.

[pradyunsg]

I don't think we should mix VCS and PEP 508 as proposed, since we'd be building on top of the standard, which would make pip non-compliant with it, in a very implementation-specific manner, which would defeat the point of the standardization. :)

[sbidoul]

This isn't a valid PEP 508 requirement

@pradyunsg how so? IIUC, PEP 508 builds on PEP 440 and the VCS form is documented in PEP 440.
The git+https form is valid and works just fine with pip. It's only the git+file form that fails.

[uranusjr]

PEP 440 specifically says the URL should point to either an sdist or wheel. It’s possible to make a case for pip to support VCS requirements like this, but that would still be pip-specific, and not covered by PEP 440.

[sbidoul]

This section of PEP 440 talks about VCS references. While it does only show git+https examples, I see nothing in it that excludes git+file references.

[sbidoul]

So do we agree this is a bug?

It comes from here in packaging:

pip/src/pip/_vendor/packaging/requirements.py

Lines 109 to 115 in 153e22a

	if parsed_url.scheme == "file":
	if urlparse.urlunparse(parsed_url) != req.url:
	raise InvalidRequirement("Invalid URL given")
	elif not (parsed_url.scheme and parsed_url.netloc) or (
	not parsed_url.scheme and not parsed_url.netloc
	):
	raise InvalidRequirement("Invalid URL: {0}".format(req.url))

The file scheme is special-cased, and git+file fails because it has no netloc.

[pradyunsg]

So do we agree this is a bug?

No...

Note that OP's packaging @ git+https://github.com/pypa/packaging is not the PEP's syntax. PEP 440 states that these URLs need a @<commit-hash> or the @<tag>#<commit-hash>. There's a note in the PEP in the same section, that explicitly mentions that the PEP's syntax "isn't quite the same as the existing VCS reference notation supported by pip".

To quote examples from PEP 440:

pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
pip @ git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d
pip @ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d

Likewise, the following works:

$ pip install "pip @ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d"
Collecting pip@ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d
  Cloning https://github.com/pypa/pip.git (to revision 1.3.1) to /private/var/folders/4d/bt0_xfx56bjfmmt2bv3r5_qh0000gn/T/pip-install-md1mmlg3/pip
  Running command git clone -q https://github.com/pypa/pip.git /private/var/folders/4d/bt0_xfx56bjfmmt2bv3r5_qh0000gn/T/pip-install-md1mmlg3/pip
  Running command git checkout -q 7a414f574bcaf1a5f5bbd4684446ffd88c740025
Building wheels for collected packages: pip
  Building wheel for pip (setup.py) ... done
  Created wheel for pip: filename=pip-1.3.1-py3-none-any.whl size=233312 sha256=854661d9ba65bc0c619e360e5659657667644a0162a8e32723b6e60c083c24d5
  Stored in directory: /private/var/folders/4d/bt0_xfx56bjfmmt2bv3r5_qh0000gn/T/pip-ephem-wheel-cache-46prpvqk/wheels/34/b7/bb/9d986aee7562ebed1d4a8afbd4814605ffe65eee6d746395fd
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-1.3.1

(sorry that it says pip too many time, I literally copy-pasted the PEP's example)

[pradyunsg]

I do think we'd want to figure out a better error message here though. :)

[pradyunsg]

And, while there's definite back-and-forth going on here about what's actually the correct behavior, thanks for exploring and spotting this edge case. :)

(also, yay 2:30am! I really should fix my body clock.)