pipenv uninstall --keep-outdated does not remove dev dependency from lockfile

[GPHemsley]

Possible blocker for #3369.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[requires]
python_version = "3.6"

[packages]
netsuite = "==0.4.*"
paramiko = "*"
requests = "*"
tenacity = "*"
toml = "*"
typing-extensions = ">=3.7.4"

[dev-packages]
cx-oracle = "<5.3"
flake8 = "*"
flake8-bugbear = "*"
flake8-tabs = "*"
mypy = "*"
pylint = "*"
vulture = "*"

With pipenv 2018.11.26:

$ pipenv uninstall --keep-outdated cx-oracle
Creating a virtualenv for this project…
Pipfile: /builds/.../.../Pipfile
Using /usr/local/bin/python (3.6.10) to create virtualenv…
created virtual environment CPython3.6.10.final.0-64 in 447ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/...-Bl5i8KOg, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Virtualenv location: /root/.local/share/virtualenvs/...-Bl5i8KOg
Uninstalling cx-oracle…
Removing cx-oracle from Pipfile…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (5aaa62)!
$ git diff
diff --git a/Pipfile b/Pipfile
index 721ce55..18a8250 100644
--- a/Pipfile
+++ b/Pipfile
@@ -15,7 +15,6 @@ toml = "*"
 typing-extensions = ">=3.7.4"
 
 [dev-packages]
-cx-oracle = "<5.3"
 flake8 = "*"
 flake8-bugbear = "*"
 flake8-tabs = "*"
diff --git a/Pipfile.lock b/Pipfile.lock
index 0e9e2b0..713608e 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "09af536c038ee38452cb07921c36f8085b681427b4a028715ff43642f9aa5945"
+            "sha256": "22585cd223de7ace2dbc5eacffca26b4bdda6d73da738dfc4cab2627b55aaa62"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -154,35 +154,35 @@
         },
         "lxml": {
             "hashes": [
-                "sha256:06d4e0bbb1d62e38ae6118406d7cdb4693a3fa34ee3762238bcb96c9e36a93cd",
-                "sha256:0701f7965903a1c3f6f09328c1278ac0eee8f56f244e66af79cb224b7ef3801c",
-                "sha256:1f2c4ec372bf1c4a2c7e4bb20845e8bcf8050365189d86806bad1e3ae473d081",
-                "sha256:4235bc124fdcf611d02047d7034164897ade13046bda967768836629bc62784f",
-                "sha256:5828c7f3e615f3975d48f40d4fe66e8a7b25f16b5e5705ffe1d22e43fb1f6261",
-                "sha256:585c0869f75577ac7a8ff38d08f7aac9033da2c41c11352ebf86a04652758b7a",
-                "sha256:5d467ce9c5d35b3bcc7172c06320dddb275fea6ac2037f72f0a4d7472035cea9",
-                "sha256:63dbc21efd7e822c11d5ddbedbbb08cd11a41e0032e382a0fd59b0b08e405a3a",
-                "sha256:7bc1b221e7867f2e7ff1933165c0cec7153dce93d0cdba6554b42a8beb687bdb",
-                "sha256:8620ce80f50d023d414183bf90cc2576c2837b88e00bea3f33ad2630133bbb60",
-                "sha256:8a0ebda56ebca1a83eb2d1ac266649b80af8dd4b4a3502b2c1e09ac2f88fe128",
-                "sha256:90ed0e36455a81b25b7034038e40880189169c308a3df360861ad74da7b68c1a",
-                "sha256:95e67224815ef86924fbc2b71a9dbd1f7262384bca4bc4793645794ac4200717",
-                "sha256:afdb34b715daf814d1abea0317b6d672476b498472f1e5aacbadc34ebbc26e89",
-                "sha256:b4b2c63cc7963aedd08a5f5a454c9f67251b1ac9e22fd9d72836206c42dc2a72",
-                "sha256:d068f55bda3c2c3fcaec24bd083d9e2eede32c583faf084d6e4b9daaea77dde8",
-                "sha256:d5b3c4b7edd2e770375a01139be11307f04341ec709cf724e0f26ebb1eef12c3",
-                "sha256:deadf4df349d1dcd7b2853a2c8796593cc346600726eff680ed8ed11812382a7",
-                "sha256:df533af6f88080419c5a604d0d63b2c33b1c0c4409aba7d0cb6de305147ea8c8",
-                "sha256:e4aa948eb15018a657702fee0b9db47e908491c64d36b4a90f59a64741516e77",
-                "sha256:e5d842c73e4ef6ed8c1bd77806bf84a7cb535f9c0cf9b2c74d02ebda310070e1",
-                "sha256:ebec08091a22c2be870890913bdadd86fcd8e9f0f22bcb398abd3af914690c15",
-                "sha256:edc15fcfd77395e24543be48871c251f38132bb834d9fdfdad756adb6ea37679",
-                "sha256:f2b74784ed7e0bc2d02bd53e48ad6ba523c9b36c194260b7a5045071abbb1012",
-                "sha256:fa071559f14bd1e92077b1b5f6c22cf09756c6de7139370249eb372854ce51e6",
-                "sha256:fd52e796fee7171c4361d441796b64df1acfceb51f29e545e812f16d023c4bbc",
-                "sha256:fe976a0f1ef09b3638778024ab9fb8cde3118f203364212c198f71341c0715ca"
-            ],
-            "version": "==4.5.0"
+                "sha256:06748c7192eab0f48e3d35a7adae609a329c6257495d5e53878003660dc0fec6",
+                "sha256:0790ddca3f825dd914978c94c2545dbea5f56f008b050e835403714babe62a5f",
+                "sha256:1aa7a6197c1cdd65d974f3e4953764eee3d9c7b67e3966616b41fab7f8f516b7",
+                "sha256:22c6d34fdb0e65d5f782a4d1a1edb52e0a8365858dafb1c08cb1d16546cf0786",
+                "sha256:2754d4406438c83144f9ffd3628bbe2dcc6d62b20dbc5c1ec4bc4385e5d44b42",
+                "sha256:27ee0faf8077c7c1a589573b1450743011117f1aa1a91d5ae776bbc5ca6070f2",
+                "sha256:2b02c106709466a93ed424454ce4c970791c486d5fcdf52b0d822a7e29789626",
+                "sha256:2d1ddce96cf15f1254a68dba6935e6e0f1fe39247de631c115e84dd404a6f031",
+                "sha256:4f282737d187ae723b2633856085c31ae5d4d432968b7f3f478a48a54835f5c4",
+                "sha256:51bb4edeb36d24ec97eb3e6a6007be128b720114f9a875d6b370317d62ac80b9",
+                "sha256:7eee37c1b9815e6505847aa5e68f192e8a1b730c5c7ead39ff317fde9ce29448",
+                "sha256:7fd88cb91a470b383aafad554c3fe1ccf6dfb2456ff0e84b95335d582a799804",
+                "sha256:9144ce36ca0824b29ebc2e02ca186e54040ebb224292072250467190fb613b96",
+                "sha256:925baf6ff1ef2c45169f548cc85204433e061360bfa7d01e1be7ae38bef73194",
+                "sha256:a636346c6c0e1092ffc202d97ec1843a75937d8c98aaf6771348ad6422e44bb0",
+                "sha256:a87dbee7ad9dce3aaefada2081843caf08a44a8f52e03e0a4cc5819f8398f2f4",
+                "sha256:a9e3b8011388e7e373565daa5e92f6c9cb844790dc18e43073212bb3e76f7007",
+                "sha256:afb53edf1046599991fb4a7d03e601ab5f5422a5435c47ee6ba91ec3b61416a6",
+                "sha256:b26719890c79a1dae7d53acac5f089d66fd8cc68a81f4e4bd355e45470dc25e1",
+                "sha256:b7462cdab6fffcda853338e1741ce99706cdf880d921b5a769202ea7b94e8528",
+                "sha256:b77975465234ff49fdad871c08aa747aae06f5e5be62866595057c43f8d2f62c",
+                "sha256:c47a8a5d00060122ca5908909478abce7bbf62d812e3fc35c6c802df8fb01fe7",
+                "sha256:c79e5debbe092e3c93ca4aee44c9a7631bdd407b2871cb541b979fd350bbbc29",
+                "sha256:d8d40e0121ca1606aa9e78c28a3a7d88a05c06b3ca61630242cded87d8ce55fa",
+                "sha256:ee2be8b8f72a2772e72ab926a3bccebf47bb727bda41ae070dc91d1fb759b726",
+                "sha256:f95d28193c3863132b1f55c1056036bf580b5a488d908f7d22a04ace8935a3a9",
+                "sha256:fadd2a63a2bfd7fb604508e553d1cf68eca250b2fbdbd81213b5f6f2fbf23529"
+            ],
+            "version": "==4.5.1"
         },
         "netsuite": {
             "hashes": [
@@ -331,13 +331,6 @@
             ],
             "version": "==19.3.0"
         },
-        "cx-oracle": {
-            "hashes": [
-                "sha256:3dfedd9538f50dee41493020c1f589e5c61835a0c8fd14f5a6c47b5919258e81"
-            ],
-            "index": "pypi",
-            "version": "==5.2.1"
-        },
         "flake8": {
             "hashes": [
                 "sha256:6c1193b0c3f853ef763969238f6c81e9e63ace9d024518edc020d5f1d6d93195",
$ pipenv sync --dev
Installing dependencies from Pipfile.lock (5aaa62)…
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!

With pipenv 2020.4.1b2:

$ pipenv uninstall --keep-outdated cx-oracle
Creating a virtualenv for this project…
Pipfile: /builds/.../.../Pipfile
Using /usr/local/bin/python3.6m (3.6.10) to create virtualenv…
created virtual environment CPython3.6.10.final.0-64 in 652ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/...-Bl5i8KOg, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator


Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/...-Bl5i8KOg
Uninstalling cx-oracle…
Removing cx-oracle from Pipfile…
Locking [dev-packages] dependencies…

Building requirements...

Resolving dependencies...

Success!
Locking [packages] dependencies…

Building requirements...

Resolving dependencies...

Success!
Updated Pipfile.lock (5aaa62)!
$ git diff
diff --git a/Pipfile b/Pipfile
index 721ce55..18a8250 100644
--- a/Pipfile
+++ b/Pipfile
@@ -15,7 +15,6 @@ toml = "*"
 typing-extensions = ">=3.7.4"
 
 [dev-packages]
-cx-oracle = "<5.3"
 flake8 = "*"
 flake8-bugbear = "*"
 flake8-tabs = "*"
diff --git a/Pipfile.lock b/Pipfile.lock
index 0e9e2b0..265670b 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "09af536c038ee38452cb07921c36f8085b681427b4a028715ff43642f9aa5945"
+            "sha256": "22585cd223de7ace2dbc5eacffca26b4bdda6d73da738dfc4cab2627b55aaa62"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -190,7 +190,7 @@
                 "sha256:50a0e03a86f09a4fa3722ffc6732d029ad58d3569f703ff52a508442ad9e9219"
             ],
             "index": "pypi",
-            "version": "==0.4.1"
+            "version": "==0.4.*"
         },
         "oauthlib": {
             "hashes": [
@@ -257,8 +257,9 @@
         },
         "requests-oauthlib": {
             "hashes": [
-                "sha256:7f71572defaecd16372f9006f33c2ec8c077c3cfa6f5911a9a90202beb513f3d",
-                "sha256:b4261601a71fd721a8bd6d7aa1cc1d6a8a93b4a9f5e96626f8e4d91e8beeaa6a"
+                "sha256:b4261601a71fd721a8bd6d7aa1cc1d6a8a93b4a9f5e96626f8e4d91e8beeaa6a",
+                "sha256:fa6c47b933f01060936d87ae9327fead68768b69c6c9ea2109c48be30f2d4dbc",
+                "sha256:7f71572defaecd16372f9006f33c2ec8c077c3cfa6f5911a9a90202beb513f3d"
             ],
             "version": "==1.3.0"
         },
@@ -322,6 +323,7 @@
                 "sha256:4c17cea3e592c21b6e222f673868961bad77e1f985cb1694ed077475a89229c1",
                 "sha256:d8506842a3faf734b81599c8b98dcc423de863adcc1999248480b18bd31a0f38"
             ],
+            "markers": "python_version >= '3.5'",
             "version": "==2.4.1"
         },
         "attrs": {
@@ -375,6 +377,7 @@
                 "sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1",
                 "sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==4.3.21"
         },
         "lazy-object-proxy": {
@@ -401,6 +404,7 @@
                 "sha256:efa1909120ce98bbb3777e8b6f92237f5d5c8ea6758efea36a473e1d38f7d3e4",
                 "sha256:f3900e8a5de27447acbf900b4750b0ddfd7ec1ea7fbaf11dfa911141bc522af0"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==1.4.3"
         },
         "mccabe": {
@@ -442,6 +446,7 @@
                 "sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367",
                 "sha256:c58a7d2815e0e8d7972bf1803331fb0152f867bd89adf8a01dfd55085434192e"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==2.6.0"
         },
         "pyflakes": {
@@ -449,6 +454,7 @@
                 "sha256:0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92",
                 "sha256:35b2d75ee967ea93b55750aa9edbbf72813e06a66ba54438df2cfac9e3c27fc8"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==2.2.0"
         },
         "pylint": {
@@ -498,7 +504,7 @@
                 "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
                 "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
             ],
-            "markers": "implementation_name == 'cpython' and python_version < '3.8'",
+            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
             "version": "==1.4.1"
         },
         "typing-extensions": {
@@ -529,6 +535,7 @@
                 "sha256:aa36550ff0c0b7ef7fa639055d797116ee891440eac1a56f378e2d3179e0320b",
                 "sha256:c599e4d75c98f6798c509911d08a22e6c021d074469042177c8c86fb92eefd96"
             ],
+            "markers": "python_version >= '3.6'",
             "version": "==3.1.0"
         }
     }
$ pipenv sync --dev
Installing dependencies from Pipfile.lock (5aaa62)…
An error occurred while installing cx-oracle==5.2.1 --hash=sha256:3dfedd9538f50dee41493020c1f589e5c61835a0c8fd14f5a6c47b5919258e81! Will try again.
Installing initially failed dependencies…
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 689, in sync
[InstallError]:       pypi_mirror=state.pypi_mirror,
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 2899, in do_sync
[InstallError]:       system=system,
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1316, in do_init
[InstallError]:       pypi_mirror=pypi_mirror,
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 905, in do_install_dependencies
[InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 800, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 707, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Collecting cx-oracle==5.2.1
[pipenv.exceptions.InstallError]:   Using cached cx_Oracle-5.2.1.tar.gz (113 kB)
[pipenv.exceptions.InstallError]:     ERROR: Command errored out with exit status 1:
[pipenv.exceptions.InstallError]:      command: /root/.local/share/virtualenvs/...-Bl5i8KOg/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-f7xdhrei/cx-oracle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-f7xdhrei/cx-oracle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r7rdhzga
[pipenv.exceptions.InstallError]:          cwd: /tmp/pip-install-f7xdhrei/cx-oracle/
[pipenv.exceptions.InstallError]:     Complete output (5 lines):
[pipenv.exceptions.InstallError]:     Traceback (most recent call last):
[pipenv.exceptions.InstallError]:       File "<string>", line 1, in <module>
[pipenv.exceptions.InstallError]:       File "/tmp/pip-install-f7xdhrei/cx-oracle/setup.py", line 170, in <module>
[pipenv.exceptions.InstallError]:         raise DistutilsSetupError("cannot locate an Oracle software " \
[pipenv.exceptions.InstallError]:     distutils.errors.DistutilsSetupError: cannot locate an Oracle software installation
[pipenv.exceptions.InstallError]:     ----------------------------------------
[pipenv.exceptions.InstallError]: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Couldn't install package: cx-oracle
 Package installation failed...

[GPHemsley]

Some other observations about differences:

• Locked version of netsuite dependency is no longer an explicit version.
• Order of hashes for requests-oauthlib has changed. (Is the order significant? Or should it be deterministic?)
• Lots of marker changes. (I assume this was an intentional change.)
• Extra lines in command output, with inconsistent formatting.

[brainwane]

Possible blocker for #3369.

I believe @techalchemy decided this is not a release blocker for the recent release. (Which is not to say it's not a bug -- just to reply on the question of whether it's a blocker.)

[GPHemsley]

Indeed, though it is a regression.

[techalchemy]

Order of hashes for requests-oauthlib has changed. (Is the order significant? Or should it be deterministic?)

Yes it should be! Between which versions did you see the changes? And more importantly, do they change between locks with the current release?

Extra lines in command output, with inconsistent formatting.

This is a big problem and one which requires some structural changes to the code I think.

Locked version of netsuite dependency is no longer an explicit version.

Is the .* getting pinned? I will look into this

pipenv uninstall --keep-outdated does not remove dev dependency from lockfile

I don't think a lot of thought went into the possibility of using --keep-outdated in this fashion but I can see why you would want to. As I mentioned on IRC last night, there is a significant gap in pipenv's current capacity for making fine-tuned adjustments to the lockfile.

I think this is an area that will need some attention.

[GPHemsley]

Locked version of netsuite dependency is no longer an explicit version.

Is the .* getting pinned? I will look into this

Filed #4278 to track separately.

[techalchemy]

Thanks. I will start a PR at some point to work on the design issues with the --keep-outdated flag

[GPHemsley]

To be clear: This command used to work in pipenv 2018.11.26 (at least insofar as it removed the requested dependency from the lockfile). So from my perspective, this is a regression, not a feature request.

Can you elaborate on whether this regression was a side effect of some other intentional change?

I think we discussed on IRC last night the possibility of using pipenv uninstall without the --keep-outdated flag; I haven't yet tested that with 2020.5.28. Can you confirm whether you expect that to work?

[GPHemsley]

I think we discussed on IRC last night the possibility of using pipenv uninstall without the --keep-outdated flag; I haven't yet tested that with 2020.5.28. Can you confirm whether you expect that to work?

Confirmed that pipenv uninstall without --keep-outdated removes the dependency from the lockfile.

[GPHemsley-RELX]

Any update here? I'm getting bit by a recent change to mypy because I can't use pipenv uninstall --keep-outdated.

[azundo]

So I've resorted to removing the dependency from the lockfile using pipenv internals. Install pipenv so it's available to be imported in your virtualenv and then from within a repl, something like:

from pipenv.project import Project
project = Project()
lockfile = project.get_or_create_lockfile()
del lockfile.default["package_name"]   # if in normal dependencies
del lockfile.develop["package_name"]  # if in dev
lockfile.write()

[badench]

Is there any update on this? It's biting me right now.

[matteius]

-keep-outdated has been deprecated for removal in the main branch, the warning will be included in the next release. The reason why is a fundamental design flaw that -keep-outdated tries to merge the results of the old resolver result with any new packages from a new resolver result which often creates more problems than it solves. Guidance is to set appropriate specifiers in your Pipfile so as all of the categories get resolved to your liking. There is really no way to improve -keep-outdated implementation without pipenv writing its own resolver, but that is not in the plan as there are many benefits to continuing to utilize the pip resolver.