Underscore in package name confuses version identification, prevents hash-based upgrade

[nedbat]

I made a sample repo called version_dummy for testing installation and upgrade. If I use #egg=version_dummy, a new version specified with a new git hash will not be installed. If I use #egg=versiondummy, then it will.
To reproduce, run this shell script:

pip --version
echo
echo "*** version_dummy ***"
echo "git+https://github.com/nedbat/version_dummy@da8cfc943a8acb3140349a81e96ea52430bf0e02#egg=version_dummy" > requirements.txt
pip install -r requirements.txt
python -c "import version_dummy as v; print '* Version dummy is', v.VERSION"
echo "git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=version_dummy" > requirements.txt
pip install -r requirements.txt
python -c "import version_dummy as v; print '* Version dummy is', v.VERSION"
pip uninstall -y version_dummy

echo
echo "*** version_dummy with mismatched name ***"
echo "git+https://github.com/nedbat/version_dummy@da8cfc943a8acb3140349a81e96ea52430bf0e02#egg=versiondummy" > requirements.txt
pip install -r requirements.txt
python -c "import version_dummy as v; print '* Version dummy is', v.VERSION"
echo "git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=versiondummy" > requirements.txt
pip install -r requirements.txt
python -c "import version_dummy as v; print '* Version dummy is', v.VERSION"
pip uninstall -y version_dummy

I get this output:

pip 7.1.2 from /usr/local/virtualenvs/pipmys/lib/python2.7/site-packages (python 2.7)

*** version_dummy ***
Collecting version-dummy from git+https://github.com/nedbat/version_dummy@da8cfc943a8acb3140349a81e96ea52430bf0e02#egg=version_dummy (from -r requirements.txt (line 1))
  Cloning https://github.com/nedbat/version_dummy (to da8cfc943a8acb3140349a81e96ea52430bf0e02) to /private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/pip-build-soigVC/version-dummy
  Could not find a tag or branch 'da8cfc943a8acb3140349a81e96ea52430bf0e02', assuming commit.
Installing collected packages: version-dummy
  Running setup.py install for version-dummy
Successfully installed version-dummy-1.0
* Version dummy is 1.0
Requirement already satisfied (use --upgrade to upgrade): version-dummy from git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=version_dummy in /usr/local/virtualenvs/pipmys/lib/python2.7/site-packages (from -r requirements.txt (line 1))
* Version dummy is 1.0
Uninstalling version-dummy-1.0:
  Successfully uninstalled version-dummy-1.0

*** version_dummy with mismatched name ***
Collecting versiondummy from git+https://github.com/nedbat/version_dummy@da8cfc943a8acb3140349a81e96ea52430bf0e02#egg=versiondummy (from -r requirements.txt (line 1))
  Cloning https://github.com/nedbat/version_dummy (to da8cfc943a8acb3140349a81e96ea52430bf0e02) to /private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/pip-build-SLdc2H/versiondummy
  Could not find a tag or branch 'da8cfc943a8acb3140349a81e96ea52430bf0e02', assuming commit.
Installing collected packages: versiondummy
  Running setup.py install for versiondummy
Successfully installed versiondummy
* Version dummy is 1.0
Collecting versiondummy from git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=versiondummy (from -r requirements.txt (line 1))
  Cloning https://github.com/nedbat/version_dummy (to ca10cfd07afdce20d61249d5a8e306cf64a3b2b5) to /private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/pip-build-XqKOaV/versiondummy
  Could not find a tag or branch 'ca10cfd07afdce20d61249d5a8e306cf64a3b2b5', assuming commit.
Installing collected packages: versiondummy
  Running setup.py install for versiondummy
Successfully installed versiondummy
* Version dummy is 2.0
Uninstalling version-dummy-1.0:
  Successfully uninstalled version-dummy-1.0

[nedbat]

Related to, or dupe of, #1249

[xavfernandez]

I'd say the misnamed one works because pip checks whether a versiondummy is already installed and since it never finds any (since the correct name is version_dummy), always performs the install.

A solution could be to go with @rbtcollins proposal and always reinstall when provided with a direct link, cf #536 (comment)

[xavfernandez]

Also note that the second case of misnamed egg, which ends up installing version 1 and 2 simultaneously won't happen anymore in pip 8 since #3153.

[pradyunsg]

@pypa/pip-committers Is this still something where we want to take action?

[xavfernandez]

I think #4764 will also fix this.

[gutsytechster]

I tried to reproduce it with the latest pip version pip 20.1 by running the above script(formatted to python3 syntax by changing the print statement to print() method). That generated the output as

pip 20.1 from /home/gutsytechster/Envs/test-pip/lib/python3.6/site-packages/pip (python 3.6)

*** version_dummy ***
Processing /home/gutsytechster/.cache/pip/wheels/72/70/6e/849d383dddcd8f0c3310f01f06af397fdabde64dc063fda79c/version_dummy-1.0-py3-none-any.whl
Installing collected packages: version-dummy
Successfully installed version-dummy-1.0
* Version dummy is 1.0
Requirement already satisfied: version_dummy from git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=version_dummy in /home/gutsytechster/Envs/test-pip/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.0)
* Version dummy is 1.0
Found existing installation: version-dummy 1.0
Uninstalling version-dummy-1.0:
  Successfully uninstalled version-dummy-1.0

*** version_dummy with mismatched name ***
Collecting versiondummy
  Cloning https://github.com/nedbat/version_dummy (to revision da8cfc943a8acb3140349a81e96ea52430bf0e02) to /tmp/pip-install-5sv5u1up/versiondummy
  Running command git clone -q https://github.com/nedbat/version_dummy /tmp/pip-install-5sv5u1up/versiondummy
  Running command git checkout -q da8cfc943a8acb3140349a81e96ea52430bf0e02
  WARNING: Generating metadata for package versiondummy produced metadata for project name version-dummy. Fix your #egg=versiondummy fragments.
Building wheels for collected packages: version-dummy, version-dummy
  Building wheel for version-dummy (setup.py) ... done
  Created wheel for version-dummy: filename=version_dummy-1.0-py3-none-any.whl size=1262 sha256=fba3e8005a1bf25785395f12b37e03022937bc527ee5d9d5dcdf1ad0fa1a3387
  Stored in directory: /home/gutsytechster/.cache/pip/wheels/72/70/6e/849d383dddcd8f0c3310f01f06af397fdabde64dc063fda79c
  Building wheel for version-dummy (setup.py) ... done
  Created wheel for version-dummy: filename=version_dummy-1.0-py3-none-any.whl size=1262 sha256=66a13e635d8202c1988cb430a3edeadfc9d2535515269946269176dcec4b6def
  Stored in directory: /tmp/pip-ephem-wheel-cache-p3hl7o7x/wheels/d8/a8/fe/59397ede9a2f8cfe782897a46bba8e3491f3df9477865ff91b
Successfully built version-dummy version-dummy
Installing collected packages: version-dummy
Successfully installed version-dummy-1.0
* Version dummy is 1.0
Collecting versiondummy
  Cloning https://github.com/nedbat/version_dummy (to revision ca10cfd07afdce20d61249d5a8e306cf64a3b2b5) to /tmp/pip-install-k8c8b304/versiondummy
  Running command git clone -q https://github.com/nedbat/version_dummy /tmp/pip-install-k8c8b304/versiondummy
  Running command git checkout -q ca10cfd07afdce20d61249d5a8e306cf64a3b2b5
  WARNING: Generating metadata for package versiondummy produced metadata for project name version-dummy. Fix your #egg=versiondummy fragments.
Requirement already satisfied (use --upgrade to upgrade): version-dummy from git+https://github.com/nedbat/version_dummy@ca10cfd07afdce20d61249d5a8e306cf64a3b2b5#egg=versiondummy in /home/gutsytechster/Envs/test-pip/lib/python3.6/site-packages (from -r requirements.txt (line 1))
Building wheels for collected packages: version-dummy, version-dummy
  Building wheel for version-dummy (setup.py) ... done
  Created wheel for version-dummy: filename=version_dummy-2.0-py3-none-any.whl size=1260 sha256=fe7a70f936e8bc99da437a15ee4b7e5f73a2315890862ce53e3795bcccc80cfe
  Stored in directory: /home/gutsytechster/.cache/pip/wheels/b2/b7/d7/754bd4432fd2b4a8349a75922cad14bd1ebbe2ff9bfe1446bf
  Building wheel for version-dummy (setup.py) ... done
  Created wheel for version-dummy: filename=version_dummy-2.0-py3-none-any.whl size=1260 sha256=fe7a70f936e8bc99da437a15ee4b7e5f73a2315890862ce53e3795bcccc80cfe
  Stored in directory: /tmp/pip-ephem-wheel-cache-md6fn1s_/wheels/1c/ff/39/138ab766be7f12cdd9bb92f49edac49ca4bd50f7344a869606
Successfully built version-dummy version-dummy
* Version dummy is 1.0
Found existing installation: version-dummy 1.0
Uninstalling version-dummy-1.0:
  Successfully uninstalled version-dummy-1.0

[DiddiLeija]

Can somebody try the latest pip version (21.1.4) to see if this is still happening?

[uranusjr]

I believe this is fixed for a while.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.