virtualenv should also copy shared libraries?

[woosley]

see this patch for pyenv

pyenv/pyenv@2657f10

When compile python with '--enable-shared -Wl,-rpath='$$ORIGIN/../lib', the created python can be dropped to any folder.

But after virtualenv copy the python binary to a different folder, the relative path "bin/../lib" to libpython.so will be broken, the python binary will die with

/root/test/bin/python3.5: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

In this case, virtualenv should also consider copy the shared object file to the new folder.

[yan12125]

I've just installed pyenv and reverted pyenv/pyenv@afbd54d, but there's still no rpath set. How to install python with rpath?

[yyuu]

It'll set rpath if PYTHON_CONFIGURE_OPTS includes --enable-shared.

https://github.com/yyuu/pyenv/blob/master/plugins/python-build/bin/python-build#L1914

[yan12125]

OK reproduced.

By the way, if you have no need on Python < 3.3, I suggest the standard venv library, which is much better in handling such cases.

[woosley]

I do need Python < 3.3.
What is the best approach for this issue? maybe an option --with-shared? if so, I can make a patch.

[yan12125]

I bet it would be better to have things auto-detected.

[woosley]

this is a bit tricky
the only solid way to detect $ORIGIN is readelf

[root@maddog 2.7.9]# readelf -d  bin/python|grep  RPATH
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib]

I am wondering in what case this readelf will be not available

[yan12125]

IMO always copying libpython.so is the file is there sounds fine. But I'm not core virtualenv developer, and I don't have merge permissions. My opinions are definitely unreliable.

[gst]

Hi,

I'm in the need of this feature as well,

any idea when it might be applied / available ?

Thx.

[gst]

@woosley :

the only solid way to detect $ORIGIN is readelf

what about :

$ ~/.pythonz/pythons/CPython-2.7.12/bin/python -c "
import sysconfig
ldflags = sysconfig.get_config_var('LDFLAGS')
print
print ldflags
print '-rpath' in ldflags.split(',')"

-Wl,-rpath,'$'ORIGIN/../lib
True

?

[yan12125]

It's possible to change rpath via utilities like chrpath.

[gst]

@yan12125 : chrpath is not necessarily available when you execute virtualenv.
so I'd expect virtualenv to copy (or symlink) the necessary lib file (which must be present ofcourse) when it can detect that the python used to create a virtualenv has been compiled with such a setting/configuration.

[yan12125]

Sorry if I wasn't clear. What I wanted to say is that it's possible to use chrpath instead of LDFLAGS when building CPython, so detecting rpath from LDFLAGS is not reliable.

[gst]

Ah, I see.
well, it's maybe possible but isn't the recommanded way is to use LDFLAGS ? so that this quite important information is recorded and made available in/via the sysconfig module.. which is designed for this AFAIK ?

[yan12125]

Yep I believe almost all packagers use LDFLAGS. I'm not sure whether virtualenv should handle edge cases or not. I'm not a maintainer but just another random person who uses virtualenv often. Those are just my two cents :)

[woosley]

Unfortunately the maintainers have been silence for a while :(

[pfmoore]

Personally, I don't have the Unix skills to comment here, sorry.

But on the subject of edge cases, I will say that one person's edge case is another person's core usage, so I'd be reluctant to make a fix for one issue that could break other users. After all, the original issue here is just as much a non-standard build of Python as anything else.

From what I can understand of the original issue here, you're building a Python installation that is relocatable (i.e., you can copy it somewhere else). If that's the case, I'm not entirely sure why you even need virtualenv - instead of using virtualenv, just take another copy of your Python installation. Virtualenv is essentially a tool that creates a partial copy of "just enough" of the Python installation, and it's pretty fragile because core Python (prior to the introduction of the venv module didn't actually support doing this. So we keep hitting issues where distributions and/or projects like pyvenv make their own custom Python builds with changes that break our approach. It's hard to solve these issues without introducing other issues elsewhere.

[gst]

giving my 2 cents:

in any case : I think that copying that .so lib to the target virtualenv (under the usual lib directory) would not break anything which would otherwise correctly behaves actually. ofcourse that would have to be somehow validated (by some test case I guess).

@pfmoore for your remark : effectively copying a truly relocatable python installation could be all good here (though we would have to wipe out site-packages to be sure to start from a fresh/clean state). I agree.
But the fact is that : even if that can be, why couldn't it (that not so special python installation) be also usable with virtualenv (especially if all it needs to is to copy (or symlink) a single extra file) ?
The virtualenv has also the added advantage of defaulting to symlink for most files..

[woosley]

I agree with what @gst said.

• I don't think it will break anything
• there will be benifits in other cases if virtualenv can add this feature. For examaple, virtualenv comes with handy tool to activate and deactive the environments. If I just relocate the installation, I have to setup all those tools myself again.

[gst]

For any info :
I've updated the PR #1045 with little enh. (and added a test case).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.