MRG, MAINT: Try conda-forge #8046

larsoner · 2020-07-22T16:20:34Z

Different approach to #7946 (just replace entirely; cc @hoechenberger ). Let's see what breaks! Locally at least I can create a seemingly-working env from this. It still only gets qt 5.12 on Linux which is a bummer, but might be good enough.

Wait for Mayavi to release 4.7.2
Wait for conda-forge to update to 4.7.2
Move everything out of pip section

larsoner · 2020-07-22T16:32:57Z

If this works, I can also take the channels: - conda-forge/label/vtk_dev from server_environment.yml to get it so that the only three things left in pip are mne/mayavi/PySurfer, which are stuck until mayavi updates. (We currently list pysurfer and mayavi in our requirements in the conda-forge recipe).

cbrnr · 2020-07-22T17:50:57Z

PyQt5 is still problematic if we don't move all packages that depend on pyqt to conda. Currently, it seems like at least pyvistaqt installed via pip will have a PyQt5 dependency, which will then overwrite whatever the pyqt package from conda installed previously. Besides creating conda-forge packages for these remaining packages (which would be the best solution), are there any alternative solutions to this problem?

cbrnr · 2020-07-22T17:51:52Z

One alternative we should try is a pip only environment. In my experience, this should have a pretty high chance to work out of the box.

larsoner · 2020-07-22T18:15:33Z

PyQt5 is still problematic if we don't move all packages that depend on pyqt to conda. Currently, it seems like at least pyvistaqt installed via pip will have a PyQt5 dependency, which will then overwrite whatever the pyqt package from conda installed previously.

I don't think so -- removing the pip part entirely and doing:

$ conda env create -n test -f environment.yml -v
...
$ conda activate test
$ pip install pyqt5
Requirement already satisfied: pyqt5 in /home/larsoner/anaconda3/envs/test/lib/python3.8/site-packages (5.12.3)

larsoner · 2020-07-22T18:16:43Z

pyqt                      5.12.3           py38ha8c2ead_3    conda-forge
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtchart                 5.12                     pypi_0    pypi
pyqtwebengine             5.12.1                   pypi_0    pypi

No idea why those other components are from pypi...

cbrnr · 2020-07-22T18:23:16Z

I don't think that's true:

$ conda create -n test pyqt
$ conda activate test
$ pip install pyqt5
Collecting pyqt5
  Using cached PyQt5-5.15.0-5.15.0-cp35.cp36.cp37.cp38-abi3-macosx_10_6_intel.whl (55.3 MB)
Collecting PyQt5-sip<13,>=12.8
  Using cached PyQt5_sip-12.8.0-cp38-cp38-macosx_10_9_x86_64.whl (63 kB)
Installing collected packages: PyQt5-sip, pyqt5
Successfully installed PyQt5-sip-12.8.0 pyqt5-5.15.0

larsoner · 2020-07-22T18:33:54Z

Indeed I get that behavior with your command. If I use conda-forge like in this PR's environment.yml:

$ conda create -c conda-forge -n test pyqt
...
$ conda activate test
$ pip install pyqt5
Requirement already satisfied: pyqt5 in /home/larsoner/anaconda3/envs/test/lib/python3.8/site-packages (5.12.3)

Can you reproduce? If so it seems like this is one (nice) difference between defaults and conda-forge provided PyQt's

hoechenberger · 2020-07-22T18:44:08Z

I can try to package pyvistaqt for conda-forge tomorrow.

Another issue is mayavi, since typically conda-forge only accepts released package versions, while we currently seem to depend on an unreleased development version

drammock · 2020-07-22T19:16:19Z

Indeed I get that behavior with your command. If I use conda-forge like in this PR's environment.yml:
$ conda create -c conda-forge -n test pyqt
...
$ conda activate test
$ pip install pyqt5
Requirement already satisfied: pyqt5 in /home/larsoner/anaconda3/envs/test/lib/python3.8/site-packages (5.12.3)
Can you reproduce? If so it seems like this is one (nice) difference between defaults and conda-forge provided PyQt's

I see what @larsoner sees here.

cbrnr · 2020-07-23T07:05:43Z

environment.yml

@@ -1,6 +1,7 @@
 name: mne
 channels:
- defaults
+- conda-forge/label/vtk_dev


Why this channel? vtk is in conda-forge with its latest release version 9.0.1.

Yes, this shouldn't be needed anymore.

I agree it seems like it shouldn't be necessary but in practice it was necessary to get 3.8 to install

Maybe we can try to figure out why - the fewer channels we use the less problems we will run into. Ideally, we should only use the conda-forge channel.

Feel free to dig into it with the conda folks, in the meantime we have been using this for the server_environment.yml for a while so I'm happy living with it until conda fixes it

cbrnr · 2020-07-23T07:12:03Z

Indeed for some reason the conda-forge pyqt package manages to sync its name with pip - if I do pip list I see a PyQt5 package installed, which is not the case in the defaults channel. Anyone know how this works?

cbrnr · 2020-07-23T07:15:01Z

@larsoner pysurfer is also in available in conda-forge. @hoechenberger it looks like the only package not available is mayavi, because we need some changes that have not been released yet. Maybe we can ask them if they are up for a new release?

cbrnr · 2020-07-23T07:15:31Z

environment.yml

-  - neo
-  - python-picard
-  - PyQt5>=5.10
+  - PySurfer


This is available in conda-forge.

See note in the code, it's because mayavi has not released a vtk9 compatible version

OK, although the versions on pip and conda are identical?

Yes, because PySurfer depends on Mayavi, if it's in the conda section it will try to pull in a bad version that segfaults.

Because the MNE recipe depends on PySurfer, same story

(Segfaults during install)

cbrnr · 2020-07-23T07:16:38Z

And while we're at it, mne is of course also available from conda-forge 😝 .

hoechenberger · 2020-07-23T07:44:59Z

And while we're at it, mne is of course also available from conda-forge 😝 .

So the way I typically used set up my MNE environment was, in fact, by simply doing

conda create -c conda-forge -n mne mne
conda activate mne
conda remove --force mne
pip install --no-deps -e ~/Development/mne-python

Obviously this doesn't work nicely anymore since the development dependencies have changed quite a bit since the last stable release, but … yeah, just wanted to share this with you. :)

cbrnr · 2020-07-23T08:02:57Z

@hoechenberger this nicely outsources everything in environment.yml to the conda-forge mne recipe. Do we want to go this way? Personally, I'd say yes.

hoechenberger · 2020-07-23T08:53:54Z

@hoechenberger this nicely outsources everything in environment.yml to the conda-forge mne recipe. Do we want to go this way? Personally, I'd say yes.

I think we'd have to decide whether this shall be used for stable releases only, or also for development versions.

For releases, it should be no problem. If there's any pip-only dependencies, we should either package them for conda-forge too or provide a super-slim environment.yml that installs them.

For development versions, where all the dependencies are more dynamic, things might be a bit trickier, I guess. conda-forge currently doesn't support nightly or development builds of software. We could, however, work around this by git-tagging master once per day or week or so, and either maintaining a separate mne-dev conda-forge package, or creating a branch on mne-feedstock, which would allow us to publish different versions of the same package (all under the mne name).

cbrnr · 2020-07-23T09:02:29Z

What about this:

Provide mne-base on conda-forge, which installs only the most important hard dependencies (e.g. numpy, scipy, matplotlib)
Provide mneon conda-forge, which depends on mne-base and installs all additional (optional) dependencies (such as mayavi, vtk, etc.)
Use an environment.yml for the current dev version

We could even try to replicate this setup with pip (i.e. for releases there are mne-base and mne packages, and for the dev version we could have a requirements.txt).

hoechenberger · 2020-07-23T09:21:51Z

What about this:

Provide mne-base on conda-forge, which installs only the most important hard dependencies (e.g. numpy, scipy, matplotlib)

Provide mneon conda-forge, which depends on mne-base and installs all additional (optional) dependencies (such as mayavi, vtk, etc.)

Use an environment.yml for the current dev version

I like this idea. I've been wanting to create an mne-base package for a long time now… Which core deps would we need – how "basic" is base? Would we include pandas as well? And matplotlib -- or matplotlib-base? (the former comes with the Qt backend, the latter doesn't)

cbrnr · 2020-07-23T09:44:44Z

I like this idea. I've been wanting to create an mne-base package for a long time now… Which core deps would we need – how "basic" is base? Would we include pandas as well? And matplotlib -- or matplotlib-base? (the former comes with the Qt backend, the latter doesn't)

We could include just the required deps as mentioned in our README (numpy and scipy), but I tend to see matplotlib as a mandatory (or at least core) dependency as well. Not sure about matploblib or matplotlib-base though, but if the Qt backend is the only difference this will affect only macOS anyway, because I assume that they include the platform default backend in matplotlib-base. So I'd include matplotlib-base because this enables plotting on all three platforms (although we often advocate for the qt backend I still think the osx backend works nicely out of the box).

So except for these big three (numpy, scipy, and matplotlib-base) I wouldn't include any other dependency. pandas is not essential IMO.

larsoner · 2020-07-23T12:00:01Z

FYI Mayavi will hopefully release in the next couple of weeks, see:

enthought/mayavi#931 (comment)

larsoner · 2020-07-24T01:15:35Z

This PR was bit by the same bug as master on the pip-pre build (#8050, numpy/numpy#16913, OpenMathLib/OpenBLAS#2728, OpenMathLib/OpenBLAS#2729) worked around the same way by setting OPENBLAS_CORETYPE="prescott" in the env vars.

So now we just want to decide if we want to go with conda-forge. I think it's worth a shot. Maybe wait on this until Mayavi releases, then everything can be conda-forge based. And if we're lucky, the OpenBLAS bug fixes will be in NumPy and SciPy as well. WDYT @cbrnr @hoechenberger @drammock ?

cbrnr · 2020-07-24T07:55:53Z

I think that's a great idea! Let's wait until Mayavi makes a new release and let's track progress in the OpenBLAS issue, and then we should be ready to go conda-forge.

What about my suggestion of splitting the release packages into mne-base and mne? Our pip package mne is already like an mne-base, but we can discuss PyPI stuff separately (this is now only about conda-forge). I would really like the idea of having two user packages, and for the dev env we could still have an environment.yml (probably also mostly based on conda-forge).

hoechenberger · 2020-07-24T07:58:05Z

@cbrnr I will create as MNE-base package before lunch today. :)

hoechenberger · 2020-07-24T13:26:17Z

@cbrnr I will create as MNE-base package before lunch today. :)

conda-forge/mne-feedstock#37

drammock · 2020-07-24T17:16:08Z

So now we just want to decide if we want to go with conda-forge. I think it's worth a shot. Maybe wait on this until Mayavi releases, then everything can be conda-forge based. And if we're lucky, the OpenBLAS bug fixes will be in NumPy and SciPy as well. WDYT @cbrnr @hoechenberger @drammock ?

Does it make sense to wait until #8009 is sorted out too? I'm wary of changing our install instructions too many times in quick succession. Perhaps that worry is more related to the suggestion of splitting off an mne-base package (which I'm not quite sold on yet). Regardless, I'd like some time to test more thoroughly on different versions of macOS (after the mayavi release of course).

larsoner · 2020-07-24T17:18:22Z

I think of mne-base as an advanced user thing, I don't think it should even show up in the advanced instructions. I think we want conda install mne to pull in the optional deps, and that we should tell people to use that version (unless they have a good reason not to).

hoechenberger · 2020-11-05T07:50:04Z

We shouldn't have to explicitly install BLAS or MKL.... there's now BLAS-based SciPy builds for Windows too

larsoner · 2020-11-05T12:22:50Z

Looks like it doesn't matter. I was trying to speed it up, but the build still appears to be 20% slower or so compared to defaults-channel items.

I could add the defaults channel and move it up the priority list but combining the two seems like asking for trouble

hoechenberger · 2020-11-05T12:23:48Z

I could add the defaults channel and move it up the priority list but combining the two seems like asking for trouble

Yes, that's always messy

larsoner · 2020-11-05T19:07:52Z

Okay I was wrong -- the pytest step is 50% slower actually. It takes ~45 minutes on this PR, and ~30 on master. For an example, the slowest test on master is:

17.10s call     mne/channels/tests/test_interpolation.py::test_interpolation_eeg[None-3e-06-True-ctol0-0.0]

And on this PR it's on a test that does not even show up in the master slowest-20 list (meaning it's < 10 seconds):

93.65s call     mne/tests/test_cov.py::test_cov_estimation_on_raw[shrunk]

This seems like an unacceptable speed loss. And without the NUM_THREADS tweaks in here, it's even worse:

114.19s call     mne/tests/test_cov.py::test_cov_estimation_on_raw[shrunk]

So that's not the problem. And timing was similar when I forced MKL:

94.07s call     mne/tests/test_cov.py::test_cov_estimation_on_raw[shrunk]

Until we sort out what's causing this speed loss, I don't think this is an acceptable performance price to pay for switching.

larsoner · 2020-11-18T20:29:02Z

Okay this one actually seems to be fixed. Unsetting the MKL_NUM_THREADS limit seems to really help the conda time, and it's no longer that much slower than master even without it (~52 min on master and ~60 min last I looked).

In theory after our recent CI tweaks this should come back happy. @cbrnr @hoechenberger are you happy with the conda-forge environment.yml?

@agramfort @drammock do you agree we should make this move? It does simplify our environment.yml, and it moves us past the PyQt5 5.9 (!) that is on defaults currently without needing to resort to pip to do it afterward (which has always been an iffy, ugly workaround).

agramfort

agreed it makes things simpler.

drammock · 2020-11-19T00:19:13Z

I guess someone should go through the install docs and make sure the new environment.yaml file doesn't require changing the instructions anywhere. I can do that tomorrow in a follow-up PR, if you don't want to wait on merging this.

larsoner · 2020-11-19T00:21:48Z

Awesome, let's see if @cbrnr and @hoechenberger have comments on the actual setup, then merge.

~~@drammock we shouldn't need to update our stable install docs to reflect any upcoming changes to environment.yml, right?~~

I guess someone should go through the install docs and make sure the new environment.yaml file doesn't require changing the instructions anywhere. I can do that tomorrow in a follow-up PR, if you don't want to wait on merging this.

Let's wait. But from a very brief look I think we do create the mne env separate from the base env so hopefully no issues.

cbrnr · 2020-11-19T06:58:40Z

Very nice! @hoechenberger for a minimal env based on conda-forge an mne-base package would be really nice. I forgot if there's already an issue/PR on that? Now that I use conda config --add channels conda-forge and conda config --set channel_priority strict I can finally get rid of the default channel completely and have conda-forge-only envs (everyone else probably already knew this)!

hoechenberger · 2020-11-19T08:15:26Z

Very nice! @hoechenberger for a minimal env based on conda-forge an mne-base package would be really nice. I forgot if there's already an issue/PR on that? Now that I use conda config --add channels conda-forge and conda config --set channel_priority strict I can finally get rid of the default channel completely and have conda-forge-only envs (everyone else probably already knew this)!

So I personally started using Miniforge, which comes with all of this preset, and already your base env is composed of conda-forge packages: https://github.com/conda-forge/miniforge

The mne-base package is not ready yet, I will pick up looking into this again soon, though.

If we're lifting the PyQt limitation now, we can also include Spyder in the environment again, and ask users to run Spyder from the MNE environment.

In other news, I've discovered that Anaconda Navigator allows to create environments from an environments file entirely through the GUI; there's no need to go to the command line. So what I imagine is that we should add Spyder to the MNE environment and adjust our installation instructions for Windows and Mac such that:

users need to download the environment file
fire up Anaconda Navigator and use the "Import" functionality to create the MNE environment
navigate to the Anaconda Navigator home/welcome screen
select the MNE environment
click on Spyder
profit!

While we're talking about environments:
I gave a workshop on Monday and Tuesday and it was totally obvious to me that when teaching MNE for data processing, I also needed to give a brief introduction to BIDS; so we used MNE-BIDSas well. I would therefore suggest to include MNE-BIDS and its (few) dependencies in our user-centric environment, too.

hoechenberger · 2020-11-19T08:17:45Z

I would therefore suggest to include MNE-BIDS and its (few) dependencies in our user-centric environment, too.

We could even include MNELAB!

cbrnr · 2020-11-19T10:19:44Z

So I personally started using Miniforge, which comes with all of this preset, and already your base env is composed of conda-forge packages: https://github.com/conda-forge/miniforge

I use Miniconda and if you set the two lines in your config and do conda install python==3.9, all packages in base will be switched to conda-forge!

The mne-base package is not ready yet, I will pick up looking into this again soon, though.

Thanks, let me know if I can help!

In other news, I've discovered that Anaconda Navigator allows to create environments from an environments file entirely through the GUI; there's no need to go to the command line. [...]

Note that Spyder now provides standalone installers for Windows and macOS. I'd prefer these separate installs, but of course this means that you have to point Spyder to the desired conda env.

I gave a workshop on Monday and Tuesday and it was totally obvious to me that when teaching MNE for data processing, I also needed to give a brief introduction to BIDS; so we used MNE-BIDSas well. I would therefore suggest to include MNE-BIDS and its (few) dependencies in our user-centric environment, too.

I didn't have time to participate - do you have a link to a recording?

I'm fine with including MNELAB as an optional package in our env file - actually, this is a great idea to let users know that there is a GUI available.

larsoner · 2020-11-19T12:37:56Z

Okay let's get this in, then @hoechenberger @cbrnr feel free to open separate issues about whether we should add spyder back, etc. For Spyder in particular you could even just open a PR with it added to the env file to see if it just works, and we'll see if it causes some packages to get held back or not.

hoechenberger · 2020-11-19T12:40:41Z

Sounds good to me. Thanks for your effort here, @larsoner

* upstream/master: MRG, MAINT: Try conda-forge (mne-tools#8046) MRG, MAINT: deduplicate definition of FIFF constants (mne-tools#8537) ENH: Add realign_raw (mne-tools#8540) CI: Use 20.04 (mne-tools#8541) fix example (mne-tools#8539) MRG, VIZ, FIX: plot_sensors title and interactivity (mne-tools#8536) MRG, MAINT: Test 3.9 (mne-tools#8533)

larsoner force-pushed the forge branch from 105763a to dd598fb Compare July 22, 2020 20:24

cbrnr reviewed Jul 23, 2020

View reviewed changes

larsoner force-pushed the forge branch 2 times, most recently from dd598fb to eca5d12 Compare July 23, 2020 21:39

larsoner force-pushed the forge branch from fcf593c to a9aeb3b Compare October 14, 2020 11:57

larsoner force-pushed the forge branch from a9aeb3b to 94cfc01 Compare November 4, 2020 22:53

MAINT: Use conda-forge

0050ba2

larsoner force-pushed the forge branch from 5d48498 to 0050ba2 Compare November 18, 2020 13:42

larsoner added 5 commits November 18, 2020 08:47

STY: Flake

1a5c7b9

FIX: Speed up

cd1a467

FIX: Restore

64f43de

FIX: Restore

7b92625

FIX: MKL?

2323771

larsoner changed the title ~~WIP, MAINT: Try conda-forge~~ MRG, MAINT: Try conda-forge Nov 18, 2020

FIX: Tol

f32198e

agramfort approved these changes Nov 18, 2020

View reviewed changes

drammock approved these changes Nov 19, 2020

View reviewed changes

cbrnr approved these changes Nov 19, 2020

View reviewed changes

larsoner merged commit 0e56673 into mne-tools:master Nov 19, 2020

larsoner deleted the forge branch November 19, 2020 12:38

MRG, MAINT: Try conda-forge #8046

MRG, MAINT: Try conda-forge #8046

Conversation

larsoner commented Jul 22, 2020 • edited

larsoner commented Jul 22, 2020

cbrnr commented Jul 22, 2020

cbrnr commented Jul 22, 2020

larsoner commented Jul 22, 2020

larsoner commented Jul 22, 2020

cbrnr commented Jul 22, 2020

larsoner commented Jul 22, 2020

hoechenberger commented Jul 22, 2020

drammock commented Jul 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbrnr commented Jul 23, 2020

cbrnr commented Jul 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbrnr commented Jul 23, 2020

hoechenberger commented Jul 23, 2020

cbrnr commented Jul 23, 2020

hoechenberger commented Jul 23, 2020

cbrnr commented Jul 23, 2020 • edited

hoechenberger commented Jul 23, 2020

cbrnr commented Jul 23, 2020

larsoner commented Jul 23, 2020

larsoner commented Jul 24, 2020

cbrnr commented Jul 24, 2020

hoechenberger commented Jul 24, 2020 • edited

hoechenberger commented Jul 24, 2020

drammock commented Jul 24, 2020

larsoner commented Jul 24, 2020

hoechenberger commented Nov 5, 2020

larsoner commented Nov 5, 2020

hoechenberger commented Nov 5, 2020

larsoner commented Nov 5, 2020

larsoner commented Nov 18, 2020

agramfort left a comment

Choose a reason for hiding this comment

drammock commented Nov 19, 2020

larsoner commented Nov 19, 2020

cbrnr commented Nov 19, 2020

hoechenberger commented Nov 19, 2020 • edited

hoechenberger commented Nov 19, 2020

cbrnr commented Nov 19, 2020

larsoner commented Nov 19, 2020

hoechenberger commented Nov 19, 2020

larsoner commented Jul 22, 2020 •

edited

cbrnr commented Jul 23, 2020 •

edited

hoechenberger commented Jul 24, 2020 •

edited

hoechenberger commented Nov 19, 2020 •

edited