Installation in sites-package missing when running ./manage.sh cython

[oliche]

When running ./manage.sh cython the extensions compile correctly but the poetry environment doesn't get recognized and the installer tries to add the package in the base environment and gets denied.

I've been around the issue by:

cd bindings/cython
pip install -e .

This may be related to poetry, just pointing out the workaround !

Full log of the manage.sh cython command

Compiling datoviz/pydatoviz.pyx because it depends on ./datoviz/cydatoviz.pxd.
[1/1] Cythonizing datoviz/pydatoviz.pyx
warning: datoviz/pydatoviz.pyx:731:37: Too few members for 'DvzAutorun'
/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages/setuptools/dist.py:519: InformationOnly: Normalizing '0.1.0.a1' to '0.1.0a1'
  self.metadata.version = self._normalize_version(
running build_ext
building 'datoviz.pydatoviz' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages/numpy/core/include -I/home/olivier/Documents/PYTHON/datoviz/include -I/opt/vulkan/x86_64/include -I/home/olivier/Documents/PYTHON/datoviz/build/_deps/cglm-src/include -I/home/olivier/Documents/PYTHON/datoviz/build/_deps/glfw-src/include -I/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/include -I/usr/include/python3.10 -c datoviz/pydatoviz.c -o build/temp.linux-x86_64-cpython-310/datoviz/pydatoviz.o -w
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/datoviz/pydatoviz.o -L/home/olivier/Documents/PYTHON/datoviz/build -L/usr/lib/x86_64-linux-gnu -ldatoviz -o build/lib.linux-x86_64-cpython-310/datoviz/pydatoviz.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-cpython-310/datoviz/pydatoviz.cpython-310-x86_64-linux-gnu.so -> datoviz
/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages/setuptools/dist.py:519: InformationOnly: Normalizing '0.1.0.a1' to '0.1.0a1'
  self.metadata.version = self._normalize_version(
running develop
/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
WARNING: The user site-packages directory is disabled.
/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
Checking .pth file support in /home/olivier/.local/lib/python3.10/site-packages
/home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/bin/python3 -E -c pass
TEST FAILED: /home/olivier/.local/lib/python3.10/site-packages does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/olivier/.local/lib/python3.10/site-packages

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.
running egg_info
writing datoviz.egg-info/PKG-INFO
writing dependency_links to datoviz.egg-info/dependency_links.txt
writing requirements to datoviz.egg-info/requires.txt
writing top-level names to datoviz.egg-info/top_level.txt
reading manifest file 'datoviz.egg-info/SOURCES.txt'
writing manifest file 'datoviz.egg-info/SOURCES.txt'
running build_ext
Creating /home/olivier/.local/lib/python3.10/site-packages/datoviz.egg-link (link to .)

Installed /home/olivier/Documents/PYTHON/datoviz/bindings/cython
Processing dependencies for datoviz==0.1.0a1
Searching for colorcet==3.0.1
Best match: colorcet 3.0.1
Installing colorcet script to /home/olivier/.local/bin

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Searching for imageio==2.29.0
Best match: imageio 2.29.0
Installing imageio_download_bin script to /home/olivier/.local/bin
Installing imageio_remove_bin script to /home/olivier/.local/bin

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Searching for numpy==1.24.3
Best match: numpy 1.24.3
Installing f2py script to /home/olivier/.local/bin
Installing f2py3 script to /home/olivier/.local/bin
Installing f2py3.10 script to /home/olivier/.local/bin

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Searching for pyct==0.5.0
Best match: pyct 0.5.0
Installing pyct script to /home/olivier/.local/bin

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Searching for Pillow==9.5.0
Best match: Pillow 9.5.0

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Searching for param==1.13.0
Best match: param 1.13.0

Using /home/olivier/.cache/pypoetry/virtualenvs/iblenv-Wd2J5L8p-py3.10/lib/python3.10/site-packages
Finished processing dependencies for datoviz==0.1.0a1

[djhoese]

I think these lines:

datoviz/manage.sh

Lines 99 to 100 in 9b934e6

	python3 setup.py build_ext -i && \
	python3 setup.py develop --user && \

Need to be replaced with a single:

python3 -m pip install -e . --user

Although the --user seems very opinionated for a generic management script...but then again maybe that's the point. This --user is your main issue @oliche it seems. It isn't installing it into the base environment but rather the "shared" user space in ~/.local/.

[rossant]

@djhoese so can we actually replace by python3 -m pip install -e . without the --user? I admit I'm not sure why this is here. This part of Python has always been extremely confusing to me.

[djhoese]

If the assumption is that the user is using a system python then --user is needed as otherwise it would try to install the package in a root-owned location and potentially break the system.

If we can assume that users are using virtual environments or conda environments then removing the --user should install the package in a directory/environment that the user already owns.

The -e . is the "modern" way of doing setup.py develop and is known as an "editable install".

[djhoese]

I should note that python setup.py build_ext -i is how I do cython compilation when I'm doing quick compiling and development, but I'm not sure it is flexible in any way but I don't think pip allows for the low level control to trigger just this build step.