setup: use scikit-build for building cmatrices and cshape C extensions #311

jcfr · 2017-10-11T13:50:52Z

This commit introduces new dependencies: "scikit-built" and "cmake".

scikit-build is a drop-in replacement to setuptools.setup function
allowing to easily compile and package extensions (C/C++/Cython) by
bridging CMake and setuptools. See http://scikit-build.org

CMake is is an open-source, cross-platform family of tools designed
to build, test and package software. See https://cmake.org

Currently, scikit-build and cmake have to be explicitly (see [1]) installed
on the system. This could be done by simply doing:

pip install -U scikit-build cmake

How does it work ?

In addition to simplifying setup.py, two new file have been added:

<root>
  |
  |---- CMakeLists.txt                          (1)
  .
  .
  .
  |--- ...src
  |        |--- CMakeLists.txt                  (2)
  .
  .

The first CMakeLists.txt specifies requirements for

Python interpreter,
the associated python libraries
and a "PythonExtension" modules containing convenience CMake
functions allowing to easily build extension by encapsulating
system introspection and platform specific logic.

and include the subdirectory associated with the other CMakeLists.txt.

The second CMakeLists.txt is specific to "cmatrices" and "cshape"
extensions and it specifies:

requirement for NumPy

Then, it declares:

libraries for each extension, associate python extension specific properties
and finally add an install rule specifying where in the package
hierarchy the extension should be installed.

Et voila,

[1] Note that improvement to python packaging will be available shortly and
it will be possible to include scikit-build and cmake directly in pyproject.toml
See here for more details: https://www.python.org/dev/peps/pep-0518/

fedorov · 2017-10-11T15:20:44Z

@jcfr I tried pip install cmake just to try on a linux system, but I am getting this error:

I have pip installed with pyenv.

jcfr · 2017-10-11T19:23:29Z

@fedorov Not sure what's happening. Look like the PATH is not properly updated because it reports issue with an older version of CMake.

May you could try to run:

$ hash -r
$ pyenv rehash

Otherwise, here is what I tried to reproduce the problem ...

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Cloning into '/home/jcfr/.pyenv'...
remote: Counting objects: 15540, done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 15540 (delta 33), reused 61 (delta 27), pack-reused 15468
Receiving objects: 100% (15540/15540), 2.76 MiB | 0 bytes/s, done.
Resolving deltas: 100% (10625/10625), done.
Checking connectivity... done.
$ export PYENV_ROOT="$HOME/.pyenv"
$ export PATH="$PYENV_ROOT/bin:$PATH"
$ eval "$(pyenv init -)"
$ pyenv install 3.5.0
Downloading Python-3.5.0.tar.xz...
-> https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz
Installing Python-3.5.0...
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Installed Python-3.5.0 to /home/jcfr/.pyenv/versions/3.5.0

$ pyenv shell 3.5.0

$ pyenv versions
  system
* 3.5.0 (set by PYENV_VERSION environment variable)

$ pip install --upgrade pip
Collecting pip
  Using cached pip-9.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
Successfully installed pip-9.0.1
$ pip install cmake 
Collecting cmake
  Downloading cmake-0.8.0-cp35-cp35m-manylinux1_x86_64.whl (19.2MB)
    100% |████████████████████████████████| 19.3MB 101kB/s 
Installing collected packages: cmake
Successfully installed cmake-0.8.0
$ cmake --version 
cmake version 3.9.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).

I can also confirm shims are used:

$ which python 
/home/jcfr/.pyenv/shims/python

jcfr · 2018-02-14T14:45:46Z

@JoostJM In order to nicely support packaging and distributing pyradiomics, I just rebased this topic.

jcfr · 2018-02-14T14:46:11Z

@fedorov I will also ensure it can be installed and packaged in the SlicerRadiomics extension

JoostJM · 2018-02-14T15:30:30Z

@jcfr, It appears to be working now on Windows and Mac, but not on Linux (appears to be missing a file, I'll try to follow-up on this). However, even though it works on the Appveyor, I'm still getting the unresolved externals on my own windows machine. Could this be due to the fact that I use Visual Studio 2015 compiler? Do you have an idea why this doesn't work with skbuild, but does work with native python setuptools?

jcfr · 2018-02-14T15:32:41Z

Could this be due to the fact that I use Visual Studio 2015 compiler?

This should not be a problem

Do you have an idea why this doesn't work with skbuild, but does work with native python setuptools?

What are the steps you follow and command line you type ?

JoostJM · 2018-02-14T15:32:50Z

This is my setup output

py -2 setup.py install

setup_cmake.txt

jcfr · 2018-02-14T15:38:58Z

In which environment do you run the command to get this output ?

Anaconda, windows terminal, Visual studio terminal ...

~~And which command (e.g python setup.py bdist_wheel, python setup.py install, ...)~~

JoostJM · 2018-02-14T16:05:00Z

regular windows terminal with compiler environment variables set

This also fails when I try it in anaconda prompt

JoostJM · 2018-02-15T11:13:49Z

@jcfr, I debugged the Linux build. By default on my home Linux machine, it sort of performs a "develop" build, where the extensions are built and placed in the source directory, and in the python site-packages it just updates the easy-install.pth file. This works and PyRadiomics works without issue.

However, on CircleCI, it installs it in the site-packages, and then I discovered that skbuild ignores the package_data keyword argument in the setup() call. This argument specifies some additional files required by PyRadiomics (the parameter script validation files), which are therefore not copied and missing at runtime. This is the source of the CircleCI error in this branch.

EDIT: I also checked the Windows and Travis CI builds. They appear to have this same error (package_data is not included in the installation). The reason the tests here are not failing is because they run a python setup.py develop and not python setup.py install command. Therefore, they run the tests in the source directory, which does have the schema files directory in there, even though they are not included in any installation...

JoostJM · 2018-02-15T12:51:48Z

@jcfr, I think I found the error on Windows. On one machine it didn't work, but it did on the other and the issue (as it is so often) is the architecture of the builds. Apparently on the machine where it didn't work, the generator defaulted to Visual Studio 14.0 2015, but not Visual Studio 14.0 2015 Win64. And because my python build is 64 bits, it failed.
When I forced the correct generator (setup.py build -G "Visual Studio 14 2015 Win64"), it worked on both machines.

Is there some way to check if a generator is for 64- or 32-bits compiling? And can that be cross-checked with the target architecture?

JoostJM · 2018-03-13T11:59:30Z

@jcfr, I tried to reinstall using the current master of scikit-build, and it works now for python 2.7 on my Windows machine. However, for python3.5, there is an issue where CMake appears to find some non-existend libs and subsequently fails the build because it cannot link to those libraries. I attached the output from my commandline. The 'libraries' in question are "optimized" and "debug" (sound more like a designator, as there both an optimized and debug library are also found.
It is possible that this is not Python 3 specific, but has more to do with the fact that for python 3 I also installed the debug libraries, but did not do so for python 2.

If I go to the generated CMake files and update the project (by removing those 2 references), the project builds without error.

I run this in a general windows terminal with command:
py -3 setup.py install -G "Visual Studio 14 2015 Win64"

setup output

jcfr · 2018-03-13T12:35:52Z

Thanks for the report.

We will soon release a new version of scikit-build that should allow to successfully build pyradiomics. If I recall, this commit scikit-build/scikit-build@c3a19ba ensure datafiles are packaged as expected.

jcfr · 2018-03-13T12:36:55Z

The 'libraries' in question are "optimized" and "debug" (sound more like a designator, as there both an optimized and debug library are also found.

This should be addressed by https://github.com/scikit-build/scikit-build/pull/306/files

JoostJM · 2018-03-13T13:21:25Z

Thanks for the report.

We will soon release a new version of scikit-build that should allow to successfully build pyradiomics. If I recall, this commit scikit-build/scikit-build@c3a19ba ensure datafiles are packaged as expected.

Yes, that's why I first built scikit-build from source (current master), which already incorporates this change.

See #283 This commit introduces new dependencies: "scikit-built" and "cmake". scikit-build is a drop-in replacement to setuptools.setup function allowing to easily compile and package extensions (C/C++/Cython) by bridging CMake and setuptools. See http://scikit-build.org CMake is is an open-source, cross-platform family of tools designed to build, test and package software. See https://cmake.org Currently, scikit-build and cmake have to be explicitly (see [1]) installed on the system. This could be done by simply doing: pip install -U scikit-build cmake How does it work ? ------------------ In addition to simplifying setup.py, two new file have been added: <root> | |---- CMakeLists.txt (1) . . . |--- ...src | |--- CMakeLists.txt (2) . . The first CMakeLists.txt specifies requirements for * Python interpreter, * the associated python libraries * and a "PythonExtension" modules containing convenience CMake functions allowing to easily build extension by encapsulating system introspection and platform specific logic. and include the subdirectory associated with the other CMakeLists.txt. The second CMakeLists.txt is specific to "cmatrices" and "cshape" extensions and it specifies: * requirement for NumPy Then, it declares: * libraries for each extension, associate python extension specific properties * and finally add an install rule specifying where in the package hierarchy the extension should be installed. Et voila, [1] Note that improvement to python packaging will be available shortly and it will be possible to include scikit-build and cmake directly in pyproject.toml See here for more details: https://www.python.org/dev/peps/pep-0518/

jcfr mentioned this pull request Oct 11, 2017

setup: use scikit-build for building cmatrices and cshape C extensions #212

Closed

jcfr force-pushed the streamline-distribution-with-scikit-build-rebased branch from 1663e1d to 86b3c36 Compare October 11, 2017 14:36

JoostJM added the installation label Jan 24, 2018

jcfr force-pushed the streamline-distribution-with-scikit-build-rebased branch from 86b3c36 to 776e6f4 Compare February 14, 2018 14:44

JoostJM added this to In Progress in PyRadiomics build/installation Feb 20, 2018

JoostJM added this to the PyRadiomics 2.0 Release milestone Mar 14, 2018

JoostJM removed this from the PyRadiomics 2.0 Release milestone Jun 5, 2018

JoostJM added the postponed label Sep 20, 2018

jcfr and others added 2 commits October 5, 2018 11:15

STYL: fix Flake8 error in setup.py

8d29d41

JoostJM force-pushed the streamline-distribution-with-scikit-build-rebased branch from 2ca9bf1 to 8d29d41 Compare October 5, 2018 09:16

JoostJM force-pushed the master branch from 3095f78 to 83b510c Compare November 12, 2018 10:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setup: use scikit-build for building cmatrices and cshape C extensions #311

setup: use scikit-build for building cmatrices and cshape C extensions #311

jcfr commented Oct 11, 2017 •

edited

fedorov commented Oct 11, 2017

jcfr commented Oct 11, 2017

jcfr commented Feb 14, 2018

jcfr commented Feb 14, 2018

JoostJM commented Feb 14, 2018

jcfr commented Feb 14, 2018 •

edited

JoostJM commented Feb 14, 2018 •

edited

jcfr commented Feb 14, 2018 •

edited

JoostJM commented Feb 14, 2018 •

edited

JoostJM commented Feb 15, 2018 •

edited

JoostJM commented Feb 15, 2018 •

edited

JoostJM commented Mar 13, 2018 •

edited

jcfr commented Mar 13, 2018

jcfr commented Mar 13, 2018

JoostJM commented Mar 13, 2018

setup: use scikit-build for building cmatrices and cshape C extensions #311

Are you sure you want to change the base?

setup: use scikit-build for building cmatrices and cshape C extensions #311

Conversation

jcfr commented Oct 11, 2017 • edited

How does it work ?

fedorov commented Oct 11, 2017

jcfr commented Oct 11, 2017

jcfr commented Feb 14, 2018

jcfr commented Feb 14, 2018

JoostJM commented Feb 14, 2018

jcfr commented Feb 14, 2018 • edited

JoostJM commented Feb 14, 2018 • edited

jcfr commented Feb 14, 2018 • edited

JoostJM commented Feb 14, 2018 • edited

JoostJM commented Feb 15, 2018 • edited

JoostJM commented Feb 15, 2018 • edited

JoostJM commented Mar 13, 2018 • edited

jcfr commented Mar 13, 2018

jcfr commented Mar 13, 2018

JoostJM commented Mar 13, 2018

jcfr commented Oct 11, 2017 •

edited

jcfr commented Feb 14, 2018 •

edited

JoostJM commented Feb 14, 2018 •

edited

jcfr commented Feb 14, 2018 •

edited

JoostJM commented Feb 14, 2018 •

edited

JoostJM commented Feb 15, 2018 •

edited

JoostJM commented Feb 15, 2018 •

edited

JoostJM commented Mar 13, 2018 •

edited