Test suite: run all notebooks in one or more dirs rather than explicitly named notebooks

[willfurnass]

Currently explictly list the notebooks to be run in tox.ini. A cleaner approach would be to have

• one or more directories within which all notebooks should be run
• one or more distinct directories in which notebooks aren't automatically run

[willfurnass]

@jarmarshall @joefresna Which notebooks should/shouldn't be run by tox?

[jarmarshall]

I would say run everything (including in subfolders) in:

• docs
• TestNotebooks
• DemoNotebooks

[willfurnass]

Getting this sorted has proved more difficult than first anticipated - our CI jobs already take ~50 mins to run (Issue #366) and attempts to run additional notebooks so as to address this issue keep resulting in cell execution timeouts (see PR #363). Bump this Issue to 1.2.0?

[jarmarshall]

I'm happy with that - stick it on the second release milestone for now I think...

[willfurnass]

@jarmarshall How about we migrate all Notebooks other than the user manual and test notebooks to other repos to cut down CI job time? These auxillary repos could have their own CI and could depend on stable versions of MuMoT.

[jarmarshall]

I think this could be a sensible suggestion since it could also solve another problem, of how to publish new notebooks which are linked from readthedocs, without needing to tag a new release or have readthedocs point to the master branch to get updates?

[willfurnass]

Notebooks in separate repos could have their own mybinder links in their READMEs and have requirements.txt files referencing stable versions of MuMoT.

We'd still need to curate mybinder links in the README and Sphinx docs for MuMoT itself for Notebooks bundled with the MuMoT package.

Disadvantages of unbundling the less general notebooks:

• Not automatically tested against new MuMoT releases as pinned to specific releases
• How to make unbundled notebooks discoverable if they are spread between various repos? A central index of some sort could be useful.

@jarmarshall If you want to go ahead with this have a think about which notebooks you'd want to unbundle and how they could be distributed between repos.

[jarmarshall]

OK, this is getting overcomplicated IMO - maybe we should revert to keeping the notebooks as is... but if those to be run as tests need to be explicitly included, we could just exclude those that take a long time / are redundant...

[willfurnass]

Hmm, with that approach, Notebooks not run during MuMoT CI jobs won't ever be tested against newer versions of MuMoT. If some Notebooks are moved to separate repos then at least the CI for those repos ensures they're been tested against a known version of MuMoT, even if that version isn't the latest.