Use USERBASE based package installation into local directory #249

laundmo · 2022-08-13T18:02:01Z

This PR sets and uses USERBASE for pip installations. This has the advantage of never interacting with the systems python installation and should prevent bugs which might happend due to wrong versions of pip (shimmed by conda)

This also localises the dependencies within the addons folder to make sure permission errors don't happen.

This is after i deletd both py-slvs and pip from any python packages directories blender can see, and clicking the "install from pip" button once:

closes: #243

also i formatted the files i touched with black, which is in line with #248 so i hope its fine even before that is closed.

TODO:

[] test how well this works with .whl installation

…cher

hlorus

This did work on my end and installed the module into CAD_Sketcher/lib/python3.10/site-packages.
For some reason it cannot find the module in the same session, i had to restart blender.
Also this added a bin folder which should also be ignored by git.

hlorus · 2022-08-14T08:50:19Z

utilities/install.py

+def install_pip():
+    import ensurepip
+
+    ret_val = ensurepip._bootstrap()


Is there a reason why the protected function is called and not just bootstrap itself?

Oh i actually had that commented in a previous version but it must've gotten lost somewhere. The bootstrap function itself doesn't do anything but call _bootstrap - but it hides the exit code of the process. will add comment.

In the docs, they do warn about environment changes when using this function.

Note The bootstrapping process has side effects on both sys.path and os.environ. Invoking the command line interface in a subprocess instead allows these side effects to be avoided.

-- https://docs.python.org/3/library/ensurepip.html#ensurepip.bootstrap

This was initially achieved when using the cmd tool as before

cmd = [global_data.PYPATH, "-m", "ensurepip", "--upgrade"] return not subprocess.call(cmd)

My question being: Do we absolutely need to keep it in this process in order to maintain the environment variables such as PYTHONUSERBASE ? The warning message could hint at future environment-related bugs.

Alternatively, I would suggest using subprocess.call as done before or subprocess.run() where you can also pass environment variables using env=env_vars in case the environment inheritance is somehow compromised.

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process’ environment. It is passed directly to Popen.

laundmo · 2022-08-14T14:11:11Z

This did work on my end and installed the module into CAD_Sketcher/lib/python3.10/site-packages. For some reason it cannot find the module in the same session, i had to restart blender. Also this added a bin folder which should also be ignored by git.

I'll have to add those, the folder names depend on what Python does for different OSs so we have to gitignore all versions - i forgot it only uses the Python3x folder scheme on windows. Maybe it's better to add another folder to USERBASE so that we can gitignore that without caring about OS specifics.

As for the finding the module: I'm not sure why that's failing - I'll have to boot into my linux install to test. wierd.

hlorus · 2022-08-15T15:02:44Z

Well path and user_base seems to be right:
sys.path: ['/home/yoga/Dropbox/blend/addons/CAD_Sketcher/lib/python3.10/site-packages', '/usr/share/blender/3.2/scripts/startup', '/usr/share/blender/3.2/scripts/modules', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/lib/python3.10/site-packages', '/usr/share/blender/3.2/scripts/freestyle/modules', '/usr/share/blender/3.2/scripts/addons/modules', '/home/yoga/.config/blender/3.2/scripts/addons/modules', '/usr/share/blender/3.2/scripts/addons', '/home/yoga/.config/blender/3.2/scripts/addons', '/home/yoga/Dropbox/blend/addons', '/usr/share/blender/3.2/scripts/addons_contrib']

site.USER_BASE: /home/yoga/Dropbox/blend/addons/CAD_Sketcher

hlorus · 2022-08-15T15:14:57Z

Calling importlib.invalidate_caches seems to do the trick on my end.

amrsoll

These changes do not seem applicable to my platform (linux). Restarting Blender doesn't fix the issue either

amrsoll · 2022-08-22T08:38:15Z

utilities/install.py

+    if subprocess.call(cmd, env=global_data.env_vars) == 0:
+        return True
+    return False


Suggested change

if subprocess.call(cmd, env=global_data.env_vars) == 0:

return True

return False

return subprocess.call(cmd, env=global_data.env_vars) == 0

Also this doesn't seem to function. In the terminal stdout/err I get No module named pip but the return value is 0. Which is very strange because it should return 1 in that case. In turn, it doesn't want to install pip.

# Running with the debugger, and tried this right before executing line 40. >>> subprocess.run(cmd, env=global_data.env_vars, capture_output=True) CompletedProcess(args=['/usr/bin/python3.10', '-m', 'pip', '--version'], returncode=0, stdout=b'', stderr=b'No module named pip\n')

I do run on Linux. Given the difference for our platforms, I will try to investigate myself

As a fix, I would recommend to always use ensurepip as it will, at most, update pip to the latest version in case pip is already present.

python -m ensurepip

This invocation will install pip if it is not already installed, but otherwise does nothing.

amrsoll · 2022-08-22T08:52:11Z

utilities/install.py

+def install_pip():
+    import ensurepip
+
+    ret_val = ensurepip._bootstrap()


In the docs, they do warn about environment changes when using this function.

Note The bootstrapping process has side effects on both sys.path and os.environ. Invoking the command line interface in a subprocess instead allows these side effects to be avoided.

-- https://docs.python.org/3/library/ensurepip.html#ensurepip.bootstrap

This was initially achieved when using the cmd tool as before

cmd = [global_data.PYPATH, "-m", "ensurepip", "--upgrade"] return not subprocess.call(cmd)

My question being: Do we absolutely need to keep it in this process in order to maintain the environment variables such as PYTHONUSERBASE ? The warning message could hint at future environment-related bugs.

Alternatively, I would suggest using subprocess.call as done before or subprocess.run() where you can also pass environment variables using env=env_vars in case the environment inheritance is somehow compromised.

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process’ environment. It is passed directly to Popen.

amrsoll · 2022-08-22T09:00:23Z

utilities/install.py

+    if no_deps:
+        cmd += ["--no-deps"]
+
+    cmd = cmd + package.split(" ")


This isn't strictly your change, but the .split(" ") will break file paths containing spaces. If we are parsing some input value, it should be done at the source View3D_OT_slvs_install_package.execute()

and you can expect file paths if we are installing .whl wheels from a different folder.

amrsoll · 2022-08-22T09:32:27Z

utilities/install.py

+    # code which is needed for the currect error messages.
+    ret_val = ensurepip._bootstrap()


Typo + link to ref

Suggested change

# code which is needed for the currect error messages.

ret_val = ensurepip._bootstrap()

# code which is needed for the correct error messages.

# https://github.com/python/cpython/blob/f4c03484da59049eb62a9bf7777b963e2267d187/Lib/ensurepip/__init__.py#L115-127

ret_val = ensurepip._bootstrap()

Cheaterman · 2023-02-14T12:33:35Z

The Linux issues here are fairly interesting, and indeed need to be solved ; thankfully I'm on Linux too, but I'll need someone to take a look at my fixed version later on other platforms, to make sure I didn't break anything. :-)

alexfanqi · 2023-03-16T11:22:13Z

The linux issue might be py_slv 1.0.5 wheel package's problem. realthunder/slvs_py#7
somehow, this specific version installs module files into wrong location. The 1.0.3 wheel version works fine even without this pr.

laundmo added 3 commits August 13, 2022 20:00

ensurepip through import, USERSITE env if possible

7f20fe3

Merge branch 'use-correct-pip' of https://github.com/laundmo/CAD_Sket…

2122516

…cher

remove unreachable

331f49f

laundmo marked this pull request as draft August 13, 2022 18:03

USERBASE based local installation of py-slvs

a04be38

laundmo changed the title ~~Use blender or usersite pip installation, not system~~ Use USERBASE based package installation into local directory Aug 14, 2022

laundmo marked this pull request as ready for review August 14, 2022 02:09

hlorus reviewed Aug 14, 2022

View reviewed changes

comments and documentation

f3567e6

amrsoll requested changes Aug 22, 2022

View reviewed changes

amrsoll reviewed Aug 22, 2022

View reviewed changes

hlorus mentioned this pull request Mar 12, 2023

[BUG] No module named 'py_slvs' #363

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use USERBASE based package installation into local directory #249

Use USERBASE based package installation into local directory #249

laundmo commented Aug 13, 2022 •

edited

hlorus left a comment

hlorus Aug 14, 2022

laundmo Aug 14, 2022

amrsoll Aug 22, 2022

laundmo commented Aug 14, 2022 •

edited

hlorus commented Aug 15, 2022 •

edited

hlorus commented Aug 15, 2022

amrsoll left a comment

amrsoll Aug 22, 2022

amrsoll Aug 22, 2022 •

edited

amrsoll Aug 22, 2022 •

edited

amrsoll Aug 22, 2022

amrsoll Aug 22, 2022

amrsoll Aug 22, 2022

amrsoll Aug 22, 2022 •

edited

Cheaterman commented Feb 14, 2023

alexfanqi commented Mar 16, 2023 •

edited

		# code which is needed for the currect error messages.
		ret_val = ensurepip._bootstrap()

Use USERBASE based package installation into local directory #249

Are you sure you want to change the base?

Use USERBASE based package installation into local directory #249

Conversation

laundmo commented Aug 13, 2022 • edited

hlorus left a comment

Choose a reason for hiding this comment

hlorus Aug 14, 2022

Choose a reason for hiding this comment

laundmo Aug 14, 2022

Choose a reason for hiding this comment

amrsoll Aug 22, 2022

Choose a reason for hiding this comment

laundmo commented Aug 14, 2022 • edited

hlorus commented Aug 15, 2022 • edited

hlorus commented Aug 15, 2022

amrsoll left a comment

Choose a reason for hiding this comment

amrsoll Aug 22, 2022

Choose a reason for hiding this comment

amrsoll Aug 22, 2022 • edited

Choose a reason for hiding this comment

amrsoll Aug 22, 2022 • edited

Choose a reason for hiding this comment

amrsoll Aug 22, 2022

Choose a reason for hiding this comment

amrsoll Aug 22, 2022

Choose a reason for hiding this comment

amrsoll Aug 22, 2022

Choose a reason for hiding this comment

amrsoll Aug 22, 2022 • edited

Choose a reason for hiding this comment

Cheaterman commented Feb 14, 2023

alexfanqi commented Mar 16, 2023 • edited

laundmo commented Aug 13, 2022 •

edited

laundmo commented Aug 14, 2022 •

edited

hlorus commented Aug 15, 2022 •

edited

amrsoll Aug 22, 2022 •

edited

amrsoll Aug 22, 2022 •

edited

amrsoll Aug 22, 2022 •

edited

alexfanqi commented Mar 16, 2023 •

edited