Updating Conda environment requirements

[rwest]

Motivation or Problem

Now would be a good time to check and update our environment dependencies.

Original motivation and context: PR #2321 needed to upgrade Cantera from 2.3 (resolved in #2288). Mopac was also stuck in the past (resolved in #2417). We have various issues with julia and pyjulia (#2444 tries to use conda-forge's pyjulia, #2443 tries to create a system image to accelerate time to launch). We are now running CI tests on Ubuntu and MacOS, which is good. PR #2311 is preparing for a new release and new Conda binaries.

This PR was first opened in July 2022, but the original commits and comments will be deleted as they are now outdated. We'll start again in May 2023.

Goals

Some of these goals might be conflicting, but I think we should try to be:

• working (this should be number 1)
• flexible, so there's not just one unique solution of versions. (I think this means have many binaries available)
• precise, so conda can find a solution quickly and it doesn't take long to make the environment (I think this means that if there is only one good solution, we specify it in the environment file so conda finds the solution quickly).
• delegate, so we're not responsible for maintaining and updating all the binaries (I think this means if we can use a conda-forge or official version of something, we should, rather than our own)
• cross-platform. We should test on multiple operating systems, versions, hardwares, compilers, etc.

Description of Changes

For a while, this draft PR will be a work in progress. Please contribute commits and comments. Don't be afraid to break things.

Summary

Summary of discussion made below.
Suggestions

• we should consider python>=3.7 instead of python=3.7in diffeqpy and re-release it.
• we should consider sundials >=2.6.2 and relax the python >=3.7,<3.8.0a0 in muq2 and re-release it

Testing

Reviewer Tips

[alongd]

I couldn't resolve it either w/o major modifications to the requirements. Here's a record of my trials if it's any help. I tried using mamba instead of conda.

(usage: before activating a conda env do conda install -c conda-forge mamba, then proceed as you would for managing envs, but replace any conda command with mamba, e.g: mamba env create -f environment.yml)

I got the following:

Encountered problems while solving.
Problem: package diffeqpy-1.1.0-rmg_1 requires python 3.7.*, but none of the providers can be installed

From looking at rmg::diffeqpy, I see under info/recipe/meta.yml the constraint python=3.7

Trying to solve the rmg_env knocking out the rmg::diffeqpy requirement gives:

Encountered problems while solving.
Problem: package cantera-2.5.0-py36hcbfb300_0 requires python >=3.6,<3.7.0a0, but none of the providers can be installed

changing the cantera requirement to conda-forge::cantera >=2.6 gives:

Encountered problems while solving.
Problem: package muq2-0.2.0-py37h6bb024c_0 requires sundials 2.6.2, but none of the providers can be installed

From looking at rmg::muq2, I see under info/recipe/meta.yml the constraint sundials ==2.6.2

• we should consider the implications of increasing the python version requirement in diffeqpy and re-release it. We should probably require python>=3.7 instead of python=3.7
• we should consider sundials >=2.6.2 in muq2

Note that muq2 has python >=3.7,<3.8.0a0. May or may not be an issue now, but will be an issue later.

Trying to solve the rmg_env knocking out the rmg::muq2 requirement,
The env was resolved, but we're missing diffeqpy and muq2 of course. Perhaps try uploading new versions of them to the anaconda rmg channel, relaxing the above constraints, hoping that it won't introduce additional problems...

(Comment edited by @rwest to abbreviate)

[rwest]

Didn't mean to close. Just force pushed an update, which then had no commits, and github closed the PR. Have now incorporated #2444 which switches us to the conda-forge version of pyjulia. (Hopefully that PR can be merged ahead of this one).

[rwest]

On my Mac, just creating an empty conda environment and requesting only pyrms

conda create --name rmgtest1 rmg::pyrms
Channels:
 - defaults
 - conda-forge
 - rmg
Platform: osx-64

  pyrms              rmg/osx-64::pyrms-0.1.2-no_rmg_dep 

you get the 0.1.2-no_rmg_dep
version (and a ton of other stuff)

On clean Ubuntu

$ conda create -n rmgtest1 rmg::pyrms
Channels:
 - defaults
 - rmg
Platform: linux-64

  pyrms              rmg/linux-64::pyrms-0.1.2-no_rmg_dep

you get the same 0.1.2-no_rmg_dep version (and a ton of other stuff)

But on Ubuntu after doing conda config --append channels conda-forge

conda create -n rmgtest1 rmg::pyrms
Channels:
 - defaults
 - conda-forge
 - rmg
Platform: linux-64

  pyrms              rmg/linux-64::pyrms-1.0.2-rmg_1
  rmg                rmg/linux-64::rmg-3.0.0-py37hb4e6623_0
  rmgdatabase        rmg/linux-64::rmgdatabase-3.1.0-py37_0

you get a different rmg/linux-64::pyrms-1.0.2-rmg_1 version.
(and a ton of other stuff)
Notice that it installed rmg, the database, and a different version of rms.

So Ubuntu and Mac end up resolving differently, and on ubuntu it depends in conda-forge channel is enabled.

Our environment specifies channels

conda env create --name rmgtest1 -f environment.yml 
Channels:
 - defaults
 - rmg
 - conda-forge
 - cantera
Platform: linux-64

etc. then in conda list I find (on Ubuntu)

pyrms                     1.0.2                     rmg_1    rmg
rmg                       3.1.0            py37h1515d6f_0    rmg
rmgdatabase               3.1.0                    py37_0    rmg

i.e. on ubuntu, to create the environment in which to do a developer build from source of RMG, it actually installs the binary rmg and rmgdatabase.

Whereas on my mac I get:

pyrms                     0.1.2                no_rmg_dep    rmg

and no rmg or rmgdatabase

@mjohnson541 any thoughts? Which version of rms is preferred? Can we have it both up-to-date and not-requiring-rmg?

[codecov]

Codecov Report

Merging #2322 (a9ad0d1) into main (99201fe) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2322   +/-   ##
=======================================
  Coverage   48.13%   48.13%           
=======================================
  Files         110      110           
  Lines       30626    30626           
  Branches     7988     7988           
=======================================
  Hits        14743    14743           
  Misses      14353    14353           
  Partials     1530     1530           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[rwest]

In commit 19124e7 on May 4 the CI installed

pyjulia                   0.6.1                     rmg_1    rmg
pyrms                     0.1.2                no_rmg_dep    rmg

Same in f4d7e9c on May 8
Same in 8cfbf5c on May 10
Same in fdc3c4d on May 12 (CI log)

But in c738227 on May 13 (CI log)

pyjulia                   0.6.1                     rmg_1    rmg
pyrms                     1.0.2                     rmg_1    rmg
rmg                       3.1.0            py37h1515d6f_0    rmg
rmgdatabase               3.1.0                    py37_0    rmg

None of the changes between these commits (here) have any thing to do with conda environment. So why the change? Was a new version of rms pushed to the conda channel?
....and they answer is yes.
https://anaconda.org/RMG/pyrms/files
but only for linux-64. And apparently it requests (through conda) rmg and rmgdatabase.

[mjohnson541]

@hwpang and I released a new version of pyrms right before the training. However, we forgot about this nuance. The difference between the no_rmg_dep is rooted in the fact that RMS for now requires RMG for a few features so when installed independently it needs to install RMG. So independent installs of pyrms should require rmg, but installs of RMG-Py naturally shouldn't install rmg as a dependency of pyrms. We should generate new no_rmg_dep binaries for pyrms for rmg to use.

[hwpang]

A bit of context on why we needed to release the new version of pyrms right before RMG Training: the new version of pyrms was required for the threaded sensitivity analysis to work properly. I only had time to build the linux version (for docker) back then to be in time for the training.

[JacksonBurns]

@rwest what's the status on this PR? I think many of the issues described at the top have been resolved by other PRs. We could stand to close this and make specific issues for muq2 and diffeqpy, which would now fall under the Python 3.11 transition project.

[JacksonBurns]

@rwest for the time being I am going to close this PR in favor of #2539 which is tackling many of these issues as part of fixing the conda binaries. Perhaps further simplifications will be possible after that PR is closed, in which case we could re-open this.