--relocatable alternatives

[dpvpro]

I used the --relocatable flag. The new release 20.0 does not have this flag. How do I now create relocatable environments?

[gaborbernat]

The relocate-able flag has been always experimental, and never really worked; we no longer support it and the feature has been entirely dropped (hence the major release). Explain your use case, and we might be able to suggest an alternative approach.

[dpvpro]

CMD export branch && cd /build_dir && \
 apt-get update -y && apt-get upgrade -y && \
 pip3 install virtualenv && \
 virtualenv --always-copy --python=python3 env && \
 git clone http://user:pass@gitlab/dev/repo.git && \
 cd /build_dir/veil-repo/code/veil-common && \
 git checkout $branch && \
 python3 install.py -p /build_dir/veil-repo/code/cli-app/ -v /build_dir/env && \
 python3 install.py -p /build_dir/veil-repo/code/controller/ -v /build_dir/env && \
 python3 install.py -p /build_dir/veil-repo/code/node/ -v /build_dir/env && \
 cd /build_dir/ && \
 ./env/bin/pip3 install -r requirements.txt && \
 virtualenv --relocatable env

This is an excerpt from Dockerfile. Will the code work correctly if i remove the --relocatable flag? That is, I need to create my own isolated environment, which I can copy in any way.

[gaborbernat]

Inside a docker, I would say 90% it would. But to make sure you'd have to liaison with who maintains that docker code, might really on subtle differences of how we handled relocatable.

[JennToo]

We used this feature sometimes because our CI system would create virtualenvs in directories with long names. The name would be long enough that the #! would be too long and we can't execute the scripts in the env.

[gaborbernat]

@Nitori- for those cases it's recommended to use python -m format of invoking the tools in general 🤔

[JennToo]

That's fair. Although many Python packages behave poorly and don't allow themselves to be invoked like that.

[gaborbernat]

@Nitori- even if that's the case you can always force the invocation by invoking the executables via the python interpreter within the binary folder; e.g.

{venv}/bin/python {venv}/bin/bad-bad-tool

where {venv} can be arbitrarily long 👍

[njgraham]

At my company we use the --relocatable flag in our pipelines to help us reuse a virtualenv and save time - we use GitLab, build the virtualenv in an early step, save it as an artifact and then reuse it to run a bunch of parallel tests.

We intended to pin the version of virtualenv but due to a bug we ended up pulling the latest so our pipelines broke this morning. We fixed that bug so it's pinned and everything is running now, and we'll find a way around the flag being removed. But, thought I'd note here to convey my experience even though the issue is closed and I don't expect the flag to be supported moving forward.

[gaborbernat]

With the new app data seeder creating a virtualenv takes just under 100ms, this approach is preferred in your use case.

[bersace]

We use --relocatable to bundle a venv in an rpm. virtualenv is created in $DESTDIR and finally moved in /opt/company.

[dpvpro]

This is my case.

[gaborbernat]

So would help to be able to pass in the final target upfront for paths generated?

[bersace]

@gaborbernat yes !

[bersace]

@gaborbernat still, the virtualenv must be usable during build phase, before deployment on final location.

[gaborbernat]

Not sure what's a good solution here 🤔 I am open to proposals.

[bersace]

@gaborbernat for such case, the standard behaviour is to distinguish builddir and prefix.

virtualenv has a DEST_DIR argument that may be misleading in this case. DEST_DIR is actual venv location, which is effectively a prefix (just like /usr, /usr/local, etc.). So maybe, documenting virtualenv LOCATION should make this clearer.

Then you can add a --destdir or --root option which defaults to /. This way, virtualenv can work isolated in destdir (somewhat like a chroot). I don't know how python, pip and other scripts will have to cope with this.

[gaborbernat]

AFAIK the problem is that all generated console scripts hardcode the path during creation for the shebang. There's no standard way to make these generated console scripts with shebangs in them work both during the build and during the run after deployed. That is unless after the build someone fixes the shebangs. But that's now out of scope for virtual environment creation, so out of our control.

[bersace]

@gaborbernat you mean that we should implement this in setuptools ? What shebang can be used both isolated at build time and in target location at runtime ?

[bersace]

I'm thinking of writing a virtualenv-relocate script that edit shebangs on an existing virtualenv, making it usable in its new location. Something like :

$ virtualenv-relocate /build/dir/venv /opt/company/app/venv
Rewriting shebang of bin/pip.
Rewriting shebang of bin/pip3.5.
...
$

[gaborbernat]

I don't think setuptools is a good place for this either. It's a feature that should be part of the build system that builds in location A but then deploy to location B.

[bersace]

Well, C project allows to build isolated by simply editing PATH and LD_LIBRARY_PATH. No matter of what build system you use.

[gaborbernat]

I'm not familiar with how C achieves this, maybe someone can explain, link to it? What has been happening before is here https://github.com/pypa/virtualenv/blob/legacy/virtualenv.py#L1880-L1894; we've been basically trying to make some paths relative: namely scripts and pth/egg-files.

[Gagi2k]

My repository is also using --relocatable and i'm using an script (after the relocatable virtual env was created) to copy additional libraries to the virtualenv. With this, it was possible to create virtualenv which is fully relocatable. E.g. we use this on windows and add the virtualenv to the our installer. After the installation the python3 programs are working fine with the packaged virtualenv.

Same usecase also works on mac and linux for us....

[gaborbernat]

@Gagi2k what you do there already showed the fragility of the current relocatable implementation; you needed to do some additional scripts after the env was created to make it fully relocatable. The problem is how you can make a package fully relocatable is heavily dependent on your target environment. So the idea here is that virtualenv itself gives up on trying to make its environments fully relocatable (as it cannot succeed in lot of cases)... and instead delegates the job entirely on people writing this custom scripts, that achieve this; scripts that have the knowledge of the target environment, so know exactly just what and how much changes are needed to make something relocatable.

[Gagi2k]

True, fair point.

My problem is more or less that i now need to recreate the functionality you provided before and test it on all platforms + several distros to do exactly the same as the old function.

I think i will try to stick with an older virtualenv version for now until me or someone else has the time to implement this.

[bersace]

@Gagi2k see my comment above about a virtualenv-relocate project idea. I'll be happy to cooperate on this with other. https://github.com/spotify/dh-virtualenv/ may be a good starting point.

[Gagi2k]

@bersace thx, i already copied the old code now and created a standalone python script out of it. I think i will use that for now as a drop-in-solution, but i'm happy to switch to something else in the future or contribute the scripts i have.

[bersace]

@Gagi2k would you mind to share it ?

[Gagi2k]

@bersace It's still work in progress: https://codereview.qt-project.org/c/qt/qtivi/+/290859

The whole virtualenv 20 update causes way more problems than anticipated. Reusing the old functionality to make the scripts relocatable is not a big deal, but because it is now based on venv and with that pyvenv.cfg it gets way more complicated. E.g. on windows the old virtualenv copied a lot of base py files to /lib/python (or symlinked it). Now they are not copied at all but the pyvenv just points to the original location. Once the original location gets updated your virtualenv needs to handle it and you need to hope that everything needed is still in place. The even bigger problem is https://bugs.python.org/issue39469, which makes it hard to pass it a relative location where you setup everything to make the virtualenv relocatable...

For now, i don't think it can be done (without pyvenv.cfg supporting it).

@gaborbernat I might be misusing virtualenv for what it is original intended, but what is the official way to provide your own python3 copy with your application, as you want to allow people to extend it using pip ?

[gaborbernat]

@Gagi2k I'm sure I'm missing here something; but could you not just alter the pyenv.cfg hone as part of the install phase on a given machine?

@gaborbernat I might be misusing virtualenv for what it is original intended, but what is the official way to provide your own python3 copy with your application, as you want to allow people to extend it using pip ?

virtual environments were designed to be always a reference to some python interpreter on a machine. The base assumption is that you have a fully working python environment on the machine, and you just want to have multiple separate site-packages for it. I can see some value in making the link not fully explicit (as is now with the fully explicit path), but if you're going down that path why not just go all the way and use either PyInstaller or pex to package your code with the python executable upfront, without any easy to break references?

[Gagi2k]

@gaborbernat Sure, that would work, the biggest problem is that most users are used to be able to just rename the folder after installation and it keeps working...
But i just found out that setting PYTHONHOME does the trick as well, atleast on windows i got it working without reference to the original installation.

pex looks interesting and i will look at that more closely, thx for the hint.

[bersace]

@gaborbernat still, how to build a virtualenv in a builddir and ship it in a .deb or .rpm ?

[gaborbernat]

@gaborbernat Sure, that would work, the biggest problem is that most users are used to be able to just rename the folder after installation and it keeps working...

Not sure why they expect this. This was never the case; even with the relocatable flag, it was true in a subset of the possible cases. Hence why it got axed with v20.

@gaborbernat still, how to build a virtualenv in a builddir and ship it in a .deb or .rpm ?

@bersace this depends on lot of your use case. Does the deb/rpm guarantee that python will be available in the same location on the target machine? If so just fixup the shebang paths during installation 👍

[bersace]

@gaborbernat depending on system python is enough to ensure python is on a specific location.

Modifying installed files on postinst is a very bad idea. This require to exclude all scripts from dpkg realm, thus dpkg wont touch them on upgrade. That's not acceptable.

AFIAK, the best solution would be a virtualenv-change-prefix that edit all shebangs to remove destdir, to be executed before archiving the final package.

[gaborbernat]

@bersace and what would you set the shebangs to?

[bersace]

@gaborbernat the absolute target location. e.g. #!/opt/company/app/venv/bin/python.

[gaborbernat]

That would now imply that deploying that package to a new refroot other than / is now not supported, not?

[bersace]

@gaborbernat yes, by design. Files managed by dpkg/rpm must not be moved around by users.

[bersace]

This is somewhat different from relocatable. The purpose is not to make venv relative, but to distinguish builddir (debian/build/... or %builddir) and rundir (/).

[gaborbernat]

Closing this as there's no actionable item on our side. I'd recommend continuing discussion here, or on potential projects that try to tackle this on top of virtualenv.

[bersace]

Actually, an alternative is to somewhat vendor dependencies, without versionning them.

[jackhhh]

In my case, I have to zip a python package with needed dependencies and pass it as an 'archive' to Spark on Yarn to support my program's running on a distributed Spark cluster. Although this operation can be finished by Anaconda, the enterprise software cannot be used in my company. relocatable can solve this problem in the past, but now it disappeared.

[peloton-mqiu]

In my case, I have to zip a python package with needed dependencies and pass it as an 'archive' to Spark on Yarn to support my program's running on a distributed Spark cluster. Although this operation can be finished by Anaconda, the enterprise software cannot be used in my company. relocatable can solve this problem in the past, but now it disappeared.

Hi @jackhhh , did you find a workaround for the existing use case?
We are running into the same issue.