pip install fails (compile error) with Python 3.7rc1

[pgdr]

Description

Unable to pip install(compile error: PyThreadState has no member ...) with Python 3.7rc1.

My apologies if this has been reported, but I couldn't find it in the issues (closed/open) nor any pull request for it.

There have been similar issues with pyyaml (yaml/pyyaml#126), numpy (numpy/numpy#10500), cython (cython/cython#1978), and pygame (pygame/pygame#382).

Steps/Code to Reproduce

Install Python 3.7rc1.
pip install scikit-learn

Expected Results

pip succeeds.

Actual Results

Compilation errors:

...
sklearn/cluster/_dbscan_inner.cpp: In function ‘int
__Pyx__GetException(PyThreadState*, PyObject**, PyObject**, PyObject**)’:

sklearn/cluster/_dbscan_inner.cpp:6029:24: error: ‘PyThreadState’ has no member named ‘exc_type’
     tmp_type = tstate->exc_type;
                        ^
sklearn/cluster/_dbscan_inner.cpp:6030:25: error: ‘PyThreadState’ has no member named ‘exc_value’
     tmp_value = tstate->exc_value;
                         ^
sklearn/cluster/_dbscan_inner.cpp:6031:22: error: ‘PyThreadState’ has no member named ‘exc_traceback’
     tmp_tb = tstate->exc_traceback;
                      ^
sklearn/cluster/_dbscan_inner.cpp:6032:13: error: ‘PyThreadState’ has no member named ‘exc_type’
     tstate->exc_type = local_type;
             ^
sklearn/cluster/_dbscan_inner.cpp:6033:13: error: ‘PyThreadState’ has no member named ‘exc_value’
     tstate->exc_value = local_value;
             ^
sklearn/cluster/_dbscan_inner.cpp:6034:13: error: ‘PyThreadState’ has no member named ‘exc_traceback’
     tstate->exc_traceback = local_tb;
             ^
error: Command "g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g
-fwrapv -O3 -Wall -fPIC
-I/data/install/lib/python3.7/site-packages/numpy/core/include
-I/data/install/lib/python3.7/site-packages/numpy/core/include
-I/data/install/include/python3.7m -c sklearn/cluster/_dbscan_inner.cpp -o
build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o -MMD -MF
build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o.d" failed with exit
status 1

Versions

Linux-3.10.0-862.el7.x86_64-x86_64-with-redhat-7.5-Maipo
Python 3.7.0rc1 (default, Jun 19 2018, 10:54:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
NumPy 1.14.5
SciPy 1.1.0

[rth]

Thanks for reporting this!

Does installation of the latest dev version works?

pip install https://github.com/scikit-learn/scikit-learn/archive/master.zip

Note that you would need Cython >=0.27.3 to compile it..

(Linking to #11224 for future reference)

[jnothman]

It sounds like this might be something Cython needs to fix and release. We are dependent on Cython's support for Py3.7 here.​

[rth]

Yes, according to cython/cython#1955 the minimum requirements under Python 3.7 is Cython 0.27.3 where this issue was fixed. In #11224 I can confirm that the build works fine with Cython 0.28.

The minimal Cython version in doc/developers/advanced_installation.rst should probably be updated accordingly for Python 3.7

[jnothman]

The easiest way for us to move towards Python 3.7 support is for us to use it in our daily "latest dependencies" job. But this is easiest when NumPy and SciPy are supporting Python 3.7. NumPy is producing wheels for 3.7. At the moment, SciPy isn't making builds on Python 3.7, but is working on it (scipy/scipy#8988). We will try to follow suit.

IMO, we should not worry about Python 3.7 support in v0.20.0, but deal with it in a bug fix release (0.20.1 or 0.20.2), given the limited support in SciPy and Six (benjaminp/six#241).

[rth]

IMO, we should not worry about Python 3.7 support in v0.20.0, but deal with it in a bug fix release (0.20.1 or 0.20.2),

To summarize the situation for Python 3.7,

• with master (future 0.20.0), building scikit-learn requires Cython >= 0.27.3 and the test suite passes on Linux Compatibility with Python 3.7.0b5 #11224, but there may be other issues on plaforms that were not tested (Windows, Mac OS). What remains is,
  • to add CI
  • fix DeprecationWarning: Using or importing the ABCs from 'collections' in Python 3.7 #11121
  • build wheels Add python 3.7 support MacPython/scikit-learn-wheels#5
• with v0.19.1, building scikit-learn requires Cython >=0.27.3, <0.28 (I think). I can confirm @selasley 's report in scikit-learn fails to install with python 3.7.0 on macOS 12.13.5 #11378 (comment) that compilation fails with 0.28.3 due to Build error with cython dev #10342 . When using Cython 0.27.3, there are 75 test failures with the setup from Compatibility with Python 3.7.0b5 #11224 (see full log in sklearn-py37.log) most can be ignored (white space issues), but a few are genuine.

[cbrnr]

SciPy now has binary wheels for Python 3.7 (all platforms).

[jnothman]

FWIW, #11409 identifies that we can build with the development versions of numpy, scipy and cython, and all tests pass (albeit with PendingDeprecationWarning and DeprecationWarning). It does not try to run the example gallery, though. Nor does it try building in Windows (perhaps it should).

[sam-s]

homebrew upgraded me to python 3.7.0 and I am now left without scikit-learn.
when do you think I will be able to do pip3 install scikit-learn? a week? a month? a year?
I need this information to decide whether to wait or to downgrade to 3.6.*.
Thank you very much.

[rth]

when do you think I will be able to do pip3 install scikit-learn? a week? a month? a year?

A month very roughly for the next release. Meanwhile you can either install the development version #11320 (comment) or downgrade to python 3.6.

[jnothman]

We could consider releasing 0.19.2 in a hurry just for Py3.7 support??​

[rth]

Given the amount of issues that were opened lately about it, it might be preferable indeed. It will also simplify comparisons / regression tracking between 0.19.x and 0.20.x for people using Python 3.7 in the future.

[lesteve]

If we end up doing a 0.19.2 release, here are two things to consider adding because they missed 0.19.1 for no particular good reason.

• nose hard dependency removal. [MRG+1] Remove hard dependency on nose #9670
• fix race condition in get_params. [MRG+2] remove modification of warning registry for no reason #9569 (comment)

[coulanuk]

On Ubuntu 18.04 the recommended workaround works:
pip install https://github.com/scikit-learn/scikit-learn/archive/master.zip

[bgyarfas]

@rth

A month very roughly for the next release. Meanwhile you can either install the development version #11320 (comment) or downgrade to python 3.6.

Homebrew has updated their default python installation to 3.7. Since homebrew doesn't allow for installing specific versions downgrading to 3.6 is not an easy prospect. Just an FYI

[shusso]

@bgyarfas it should be possible to downgrade from python 3.7 back to 3.6.5 in brew

brew info python3
brew switch python 3.6.5

[cbrnr]

This works only if the previous version is still cached on the local machine.

[ledona]

@bgyarfas If you don't have 3.6 cached in homebrew you can use pyenv. The following worked for me...

brew install pyenv
pyenv install 3.6.5
eval "$(pyenv init -)"
pyenv shell 3.6.5
pip install virtualenv
virtualenv venv-3.6
source venv-3.6/bin/activate

You can do this without virtualenv, but my typical setup relies on it.

[cbrnr]

I think the point is that the average Homebrew user will have Python 3.7 by default.

[rth]

I think the point is that the average Homebrew user will have Python 3.7 by default.

A minor 0.19.2 release that will support Python 3.7 is in progress.

[sam-s]

what is the timeline for the 19.2 release?
thanks!

[shoeffner]

@sam-s If you need it now, you can try installing the master branch using the latest cython version:

pip install -U cython
pip install -e git+https://github.com/scikit-learn/scikit-learn@master#egg=scikit-learn

This worked for me on Python 3.7 using pipenv.
Of course, when the release is available, you should swap from the (possibly) unstable master branch to the correct release.

[jnothman]

you could also install from the 0.19.X branch which is basically ready for release, but the people who can make the release haven't been available.

[samueldg]

Release 0.19.2 is now available on PyPI: https://pypi.org/project/scikit-learn/0.19.2/

[rth]

Closing as fixed. Please use 0.19.2 with Python 3.7.

[marco-c]

What is the plan for 0.20.X? It looks like I can build with 0.20.0, but not with 0.20.1.
The errors are like:

     sklearn/cluster/_hierarchical.cpp:24792:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
           tstate->exc_traceback = *tb;

[lesteve]

Have you tried updating Cython as mentioned in #11320 (comment)?

[marco-c]

Just updated to 0.20.2 and it's working.

[nok]

I made the experience that all versions lower than 0.20 (e.g. 0.16, 0.17, 0.18, 0.19) will break.

My tests: https://travis-ci.org/nok/sklearn-porter/builds/561765760

Are there any plans to cover these versions?

[rth]

I made the experience that all versions lower than 0.20 (e.g. 0.16, 0.17, 0.18, 0.19) will break.
Are there any plans to cover these versions?

I don't think so. Users with Python 3.7 would need to upgrade to scikit-learn 0.20.X at least.