Editable pip install is not working, but a regular pip install and `python setup.py develop` work fine #1025

dantamont · 2023-08-28T02:56:48Z

Hi all, I am trying to put together an example project that builds/installs a Python package using SWIG. I think I may have a related, but not identical issue to what was discussed in this thread. This is my folder layout:

Note that this swig folder is actually a subdirectory of a larger CMake project, so I need to invoke the CMakeLists.txt that lives two directories up in order to correctly build the target that I am depending on:

from skbuild import setup  # This line replaces 'from setuptools import setup'
from pathlib import Path

ROOT_DIR = Path(__file__).parents[2].resolve()

setup(
    name="swig_wrapper_example",
    version="0.0.1",
    description="A minimal example package (cpp version)",
    author='Me',
    license="TBD",
    packages=['swig_wrapper'],
    python_requires=">=3.7",
    cmake_source_dir=str(ROOT_DIR),
    include_package_data=False,
    zip_safe=False
)

[build-system]
requires = [
    "setuptools>=42",
    "scikit-build>=0.13",
    "cmake>=3.18",
    "ninja",
]
build-backend = "setuptools.build_meta"

Strangely, when I run pip install -e . in my swig directory, I don't get ANY output in a local swig/_skbuild directory. In contrast, running pip install . creates a _skbuild directory, which contains a cmake-install/swig_wrapper folder and all of the expected output.

This stuff also gets correctly copied into my virtual environment's local site-packages/swig_wrapper directory:

As I mentioned, the editable install doesn't even generate an _skbuild directory, so it doesn't seem like CMake is being invoked at all. I would expect the install to produce the same output files, and to place them into swig/swig_wrapper. This unfortunately does not happen.

However, when I call python setup.py develop, everything works as expected and I get all of the right build output and files installed into my local source directory. I'm not sure what is happening differently with the editable pip install.

Any ideas on what could be going on? Here is my swig/CMakeLists.txt for reference:

project(swig_wrapper)

#### Check if SWIG is installed. If so, generate python bindings for library_to_wrap
find_package(SWIG COMPONENTS python)


if(SWIG_FOUND)
    include(UseSWIG)

    set(CMAKE_SWIG_FLAGS "")
    set(CMAKE_SWIG_OUTDIR ${CMAKE_CURRENT_BINARY_DIR}/swigbuild)
    set(SWIG_USE_TARGET_INCLUDE_DIRECTORIES TRUE)
    set(SWIG_USE_SWIG_DEPENDENCIES TRUE)

    ## Create the library_to_wrap_py python library.
    set_source_files_properties(library_to_wrap_interface.i
        PROPERTIES
            CPLUSPLUS ON
    )

    swig_add_library(${PROJECT_NAME}
        LANGUAGE
            python
        SOURCES
            library_to_wrap_interface.i
    )

    target_compile_features(${PROJECT_NAME}
        PUBLIC
            cxx_std_20
    )

    cmake_path(SET MY_SOURCE_INCLUDE_DIR NORMALIZE ${CMAKE_SOURCE_DIR}/library_to_wrap/include)
    message(STATUS "Including directories at ${MY_SOURCE_INCLUDE_DIR}")
    set_target_properties(${PROJECT_NAME}
        PROPERTIES
            SWIG_INCLUDE_DIRECTORIES ${MY_SOURCE_INCLUDE_DIR}
    )

    target_link_libraries(${PROJECT_NAME}
            library_to_wrap
            Python::Module
            )

endif(SWIG_FOUND)


#### Python module installation
find_package(SWIG COMPONENTS python)

if(SWIG_FOUND)
	find_package(PythonExtensions REQUIRED)
	python_extension_module(${PROJECT_NAME})

	install(TARGETS ${PROJECT_NAME} LIBRARY DESTINATION ${PROJECT_NAME})
	install(FILES ${CMAKE_SWIG_OUTDIR}/library_to_wrap_interface.py DESTINATION ${PROJECT_NAME})
endif()

Here is also the output of pip install -e . -v:

Using pip 23.0.1 from /home/me/venvs/my_env/lib/python3.9/site-packages/pip (python 3.9)
Obtaining file:///home/me/git_repos/python_wrapping_examples/wrappers/swig
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=42
    Using cached setuptools-68.1.2-py3-none-any.whl (805 kB)
  Collecting scikit-build>=0.13
    Using cached scikit_build-0.17.6-py3-none-any.whl (84 kB)
  Collecting cmake>=3.18
    Using cached cmake-3.27.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.1 MB)
  Collecting ninja
    Using cached ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
  Collecting distro
    Using cached distro-1.8.0-py3-none-any.whl (20 kB)
  Collecting packaging
    Using cached packaging-23.1-py3-none-any.whl (48 kB)
  Collecting tomli
    Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
  Collecting wheel>=0.32.0
    Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
  Installing collected packages: ninja, cmake, wheel, tomli, setuptools, packaging, distro, scikit-build
  Successfully installed cmake-3.27.4 distro-1.8.0 ninja-1.11.1 packaging-23.1 scikit-build-0.17.6 setuptools-68.1.2 tomli-2.0.1 wheel-0.41.2

  [notice] A new release of pip is available: 23.0.1 -> 23.2.1
  [notice] To update, run: pip install --upgrade pip
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
  Running command Getting requirements to build editable
  running egg_info
  writing swig_wrapper_example.egg-info/PKG-INFO
  writing dependency_links to swig_wrapper_example.egg-info/dependency_links.txt
  writing top-level names to swig_wrapper_example.egg-info/top_level.txt
  reading manifest file 'swig_wrapper_example.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'swig_wrapper_example.egg-info/SOURCES.txt'
  Getting requirements to build editable ... done
  Running command Preparing editable metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info
  writing /tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file '/tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-aqi16y5v/swig_wrapper_example-0.0.1.dist-info'
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: swig-wrapper-example
  Running command Building editable for swig-wrapper-example (pyproject.toml)
  running editable_wheel
  creating /tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info
  writing /tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/top_level.txt
  writing manifest file '/tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file '/tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example.egg-info/SOURCES.txt'
  creating '/tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example-0.0.1.dist-info'
  creating /tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example-0.0.1.dist-info/WHEEL
  running build_py
  copied 0 files
  Editable install will be performed using a meta path finder.

  Options like `package-data`, `include/exclude-package-data` or
  `packages.find.exclude/include` may have no effect.

  adding '__editable___swig_wrapper_example_0_0_1_finder.py'
  adding '__editable__.swig_wrapper_example-0.0.1.pth'
  creating '/tmp/pip-wheel-kchf4sky/.tmp-33eki8jt/swig_wrapper_example-0.0.1-0.editable-py3-none-any.whl' and adding '/tmp/tmpfrlmz1vgswig_wrapper_example-0.0.1-0.editable-py3-none-any.whl' to it
  adding 'swig_wrapper_example-0.0.1.dist-info/METADATA'
  adding 'swig_wrapper_example-0.0.1.dist-info/WHEEL'
  adding 'swig_wrapper_example-0.0.1.dist-info/top_level.txt'
  adding 'swig_wrapper_example-0.0.1.dist-info/RECORD'
  /tmp/pip-build-env-ztg692xa/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py:349: InformationOnly: Editable installation.
  !!

          ********************************************************************************
          Please be careful with folders in your working directory with the same
          name as your package as they may take precedence during imports.
          ********************************************************************************

  !!
    wheel_obj.write_files(unpacked)
  Building editable for swig-wrapper-example (pyproject.toml) ... done
  Created wheel for swig-wrapper-example: filename=swig_wrapper_example-0.0.1-0.editable-py3-none-any.whl size=2733 sha256=e1f59b068b7b190277f79b09aff9a47ac9552bdb55d20eb8d53ebdad42bbf1a1
  Stored in directory: /tmp/pip-ephem-wheel-cache-xm9fmda4/wheels/c0/ba/70/41796496daf670e8126fac205713be6c9439becd9edc3b8c2a
Successfully built swig-wrapper-example
Installing collected packages: swig-wrapper-example
Successfully installed swig-wrapper-example-0.0.1

I switched over to a pyproject.toml driven by scikit-build-core, and while this doesn't actually copy the CMake-generated library and python files into my local project directory, it does copy them into my site-packages and the editable install seems to work. I'll leave this issue open for now in case anybody wants to weigh in on how to get my original setup to work, but I'm considering this resolved since all of my research suggests that editable mode with scikit-build is unsupported.

The text was updated successfully, but these errors were encountered:

karosc · 2023-09-01T20:26:50Z

I think I might have a similar problem. If I run python setup.py install I can see all my C-code get compiled and my python swig wrapper generated. However, if I run pip install . nothing gets compiled. I'm not sure what I'm doing wrong here.

The project I am working on is here: https://github.com/pyswmm/swmm-python/tree/dev_pyproject.toml

dantamont changed the title ~~Editable pip install is not working, but a regular pip install works fine~~ Editable pip install is not working, but a regular pip install and python setup.py develop work fine Aug 28, 2023

sjperkins mentioned this issue Aug 30, 2023

editable install does not work with scikit-build and setuptools >= 64 ratt-ru/arcae#52

Closed

Zhaoyilunnn mentioned this issue Sep 1, 2023

feat: restructure directory ScQ-Cloud/pyquafu#71

Merged

troibe mentioned this issue Oct 13, 2023

Add Python 3.12 remove 3.6, 3.7 siliconcompiler/sc-leflib#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Editable pip install is not working, but a regular pip install and `python setup.py develop` work fine #1025

Editable pip install is not working, but a regular pip install and `python setup.py develop` work fine #1025

dantamont commented Aug 28, 2023 •

edited

karosc commented Sep 1, 2023

Editable pip install is not working, but a regular pip install and python setup.py develop work fine #1025

Editable pip install is not working, but a regular pip install and python setup.py develop work fine #1025

Comments

dantamont commented Aug 28, 2023 • edited

karosc commented Sep 1, 2023

Editable pip install is not working, but a regular pip install and `python setup.py develop` work fine #1025

Editable pip install is not working, but a regular pip install and `python setup.py develop` work fine #1025

dantamont commented Aug 28, 2023 •

edited