Do not use the upstream Pipenv #290

fridex · 2018-10-08T09:26:40Z

The recent issues on upstream Pipenv caused s2i builds to always fail:


New python executable in /opt/app-root/src/.local/venvs/pipenv/bin/python3
--
  | Also creating executable in /opt/app-root/src/.local/venvs/pipenv/bin/python
  | Installing setuptools, pip, wheel...done.
  | Collecting pipenv
  | Downloading https://files.pythonhosted.org/packages/eb/64/9b2747d54f2008ac3dfe86c0b1c8ec126042726fd8a540d5208d26732701/pipenv-2018.7.1-py3-none-any.whl  (5.0MB)
  | Collecting virtualenv (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl  (1.9MB)
  | Collecting certifi (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl  (147kB)
  | Collecting pip>=9.0.1 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl  (1.3MB)
  | Collecting virtualenv-clone>=0.2.5 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
  | Collecting setuptools>=36.2.1 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py2.py3-none-any.whl  (569kB)
  | Installing collected packages: virtualenv, certifi, pip, virtualenv-clone, setuptools, pipenv
  | Found existing installation: pip 9.0.1
  | Uninstalling pip-9.0.1:
  | Successfully uninstalled pip-9.0.1
  | Found existing installation: setuptools 28.8.0
  | Uninstalling setuptools-28.8.0:
  | Successfully uninstalled setuptools-28.8.0
  | Successfully installed certifi-2018.8.24 pip-18.1 pipenv-2018.7.1 setuptools-40.4.3 virtualenv-16.0.0 virtualenv-clone-0.3.0
  | ---> Installing dependencies via pipenv ...
  | Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
  | Installing dependencies from Pipfile.lock (c00e44)...
  | Traceback (most recent call last):
  | File "/opt/app-root/src/.local/bin/pipenv", line 11, in <module>
  | sys.exit(cli())
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
  | return self.main(*args, **kwargs)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
  | rv = self.invoke(ctx)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
  | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
  | return ctx.invoke(self.callback, **ctx.params)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
  | return callback(*args, **kwargs)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
  | selective_upgrade=selective_upgrade,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1322, in do_init
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 807, in do_install_dependencies
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1375, in pip_install
  | package_name.split('--hash')[0].split('--trusted-host')[0]
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 704, in from_line
  | line, extras = _strip_extras(line)
  | TypeError: 'module' object is not callable

The upstream issue can be found at pypa/pipenv#2924 A brief summary - Pipenv relies on some logic present in pip and with a new pip release where the shared logic changed, Pipenv broke. Even though the patch to address the issue was already merged into Pipenv master today, we are still unable to build any s2i based project that is using Pipenv as a bugfix release was not not issued by upstream since Friday when this issue occurred.

As Pipenv is already packaged as an RPM it could be worth to consider adding Pipenv to the base image from Fedora repositories so s2i does not rely on the latest Pipenv that can introduce such issues.

The text was updated successfully, but these errors were encountered:

omron93 · 2018-10-08T10:52:03Z

I hit this issue too.

@mcyprian @torsava ?

pkubatrh · 2018-10-09T07:12:19Z

@torsava How about not upgrading dependencies to latest versions when installing pipenv?

hakkeroid · 2018-10-09T15:08:59Z

As the pipenv situation didn't improve after the latest release I fixed this issue for now by overriding the assemble file. If someone wants to follow along simply copy the respective assemble file of the python version you use into your project and change the function install_pipenv like so:

function install_pipenv() {
  if [ ! -z ${PIN_PIPENV_VERSION} ]; then
    pipenv_req="pipenv==${PIN_PIPENV_VERSION}"
  else
    pipenv_req="pipenv"
  fi
  if [ ! -z ${PIN_PIP_VERSION} ]; then
    pip_req="pip==${PIN_PIP_VERSION}"
  else
    pip_req="pip"
  fi
  echo "---> Installing $pipenv_req packaging tool with $pip_req ..."
  VENV_DIR=$HOME/.local/venvs/pipenv
  virtualenv_bin "$VENV_DIR"
  $VENV_DIR/bin/pip --isolated install -U $pipenv_req $pip_req
  mkdir -p $HOME/.local/bin
  ln -s $VENV_DIR/bin/pipenv $HOME/.local/bin/pipenv
}

Now provide a specific pipenv + pip version via PIN_PIP_VERSION and PIN_PIPENV_VERSION environment variables to the build container.

Note: I used the prefix PIN_ to prevent issues with naming clashes. Namely PIP_VERSION.

If you have a better idea, let me know. :)

torsava · 2018-10-10T11:28:45Z

Apologies, I'm currently swamped with a different project. How urgent is this?

GrahamDumpleton · 2018-11-27T06:02:12Z

FWIW, you can't use RPM version of pipenv. Has to be a version which is installed in the virtual environment created for the image. That is, it doesn't use system or SCL versions of Python packages.

frenzymadness · 2019-03-15T08:20:48Z

IFAIK this is no longer a problem but I agree that it might be handy for future cases to be able to pin a specific pip/pipenv version.

Could you please open a PR or should somebody else take over it?

hakkeroid · 2019-03-16T16:04:31Z

@frenzymadness My proposal isn't tested (it was good enough for my situation though) and it has to be adapted to all assemble files I guess. However, feel free to take over. Although I worry that it will take a while to get merged, as @torsava already mentioned beeing occupied elsewhere.

torsava · 2019-03-18T16:09:13Z

A PR for pinning the version would be great. If anyone can create it, we'll get it merged.

frenzymadness · 2019-05-22T09:34:42Z

I would like to implement it but I need help. I completely understand the problem and how we can solve it but, there are two copies of pip installed in the image:

one is in the main virtual environment and is used by application itself for installation of its dependencies
and the second one is in the special virtual environment created just for pipenv itself

Implementation of PIP_PIPENV_VERSION is quite simple. But what PIN_PIP_VERSION should mean? Should it has an impact on the version of pip installed in venv with pipenv only or should both pip copies should be updated to that version? How to communicate this with users?

hakkeroid · 2019-07-14T15:21:54Z

@frenzymadness (from my memories) Usually it is only required to pin the system's pip version. That is because when pipenv is installed it is (or was) conflicting with the existing pip due to some upstream changes. What pipenv is then installing inside the virtualenv shouldn't matter anymore. However, I cannot recall why I pinned pip inside pipenv as well. Maybe it was due to some strange behavior.

one is in the main virtual environment and is used by application itself for installation of its dependencies

I think this depends on the variable ENABLE_PIPENV. So if it is enabled then the application is installed inside the virtualenv created by pipenv. Otherwise the system's pip or python is used.

How to communicate this with users?

Good question. To keep it simple I suggest to only be able to pin the system's pip or pipenv. At the end I assume most users would pin pip only, because

their application requires that, or
the upstream conflict with pipenv happens.

In case of (1) if users don't use pipenv, then pinning the system's pip makes sense. If they do use pipenv though, they probably should pin pip inside of the Pipfile instead. So the system's pip is unimportant and only used to install pipenv itself. However, this is done automatically anyway.
Only in case (2), when the automatic install fails or the resulting virtualenv is broken, then they need to fiddle around with the system's pip and for that need to understand the relationship between the system's pip, pipenv and the virtualenv's pip.

I hope that helps! :)

frenzymadness · 2019-07-15T06:11:33Z

Thanks @hakkeroid for your comment. I think that we can keep this issue open until a new bug of incompatibility between pip and pipenv shows up to see what the exact problem is and how to solve it. Pipenv was under heavy development in 2018 (more than 70 releases in one year) but now it seems to be less in rush with better care about other depending projects.

In the meantime, if it'd be beneficial for anybody to have a possibility to pin a specific version of pip, feel free to comment here or send a pull request.

fridex · 2020-03-05T14:45:43Z

FYI, we proposed an alternative "micropipenv" - we use it in aicoe to tackle also this issue in #368

frenzymadness · 2020-06-09T11:02:49Z

The latest images have the possibility to use micropipenv and in this PR #387 I'll implement PIN_PIPENV_VERSION so you'll be able to pin the best version for you.

We still cannot and don't want to include pipenv to the image itself but the latest improvements should give you enough control and reproducible builds.

frenzymadness · 2020-07-13T05:19:39Z

PIN_PIPENV_VERSION is available.

omron93 mentioned this issue Oct 8, 2018

Update to new commons for generate symlink fix #291

Merged

pkubatrh assigned torsava Oct 9, 2018

frenzymadness mentioned this issue Mar 15, 2019

include pipenv at build of s2i container image rather than downloading it during use of s2i #278

Closed

torsava added the help wanted label Mar 19, 2019

frenzymadness added the needinfo Waiting for additional information label May 22, 2019

fridex mentioned this issue Mar 5, 2020

micropipenv as an alternative for pipenv for S2I images #368

Open

fridex mentioned this issue Jun 1, 2020

hotfix: Pipenv version lock #383

Merged

frenzymadness closed this as completed Jul 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not use the upstream Pipenv #290

Do not use the upstream Pipenv #290

fridex commented Oct 8, 2018

omron93 commented Oct 8, 2018

pkubatrh commented Oct 9, 2018 •

edited

hakkeroid commented Oct 9, 2018 •

edited

torsava commented Oct 10, 2018

GrahamDumpleton commented Nov 27, 2018

frenzymadness commented Mar 15, 2019

hakkeroid commented Mar 16, 2019

torsava commented Mar 18, 2019

frenzymadness commented May 22, 2019

hakkeroid commented Jul 14, 2019

frenzymadness commented Jul 15, 2019

fridex commented Mar 5, 2020

frenzymadness commented Jun 9, 2020

frenzymadness commented Jul 13, 2020

Do not use the upstream Pipenv #290

Do not use the upstream Pipenv #290

Comments

fridex commented Oct 8, 2018

omron93 commented Oct 8, 2018

pkubatrh commented Oct 9, 2018 • edited

hakkeroid commented Oct 9, 2018 • edited

torsava commented Oct 10, 2018

GrahamDumpleton commented Nov 27, 2018

frenzymadness commented Mar 15, 2019

hakkeroid commented Mar 16, 2019

torsava commented Mar 18, 2019

frenzymadness commented May 22, 2019

hakkeroid commented Jul 14, 2019

frenzymadness commented Jul 15, 2019

fridex commented Mar 5, 2020

frenzymadness commented Jun 9, 2020

frenzymadness commented Jul 13, 2020

pkubatrh commented Oct 9, 2018 •

edited

hakkeroid commented Oct 9, 2018 •

edited