Use nbmake or testbook to automate notebook testing

[jonmaddock]

One of these two tools could make testing notebooks much more straightforward, avoiding the duplicate .py scripts. The tools need to be investigated and compared with Bluemira's strategy (which I can forward). One option should be implemented to ensure that our notebooks are well-tested in a sustainable manner.

[clmould]

There were 3 options available for automating notebook testing - nbmake, testbook and jupytext (used by Bluemira). After discussion with @timothy-nunn we decided that testbook would be the best option for PROCESS

• jupytext is used by Bluemira and uses python files instead of ipynb files and this has some advantages such as python files being easier to read as a diff, however we want to keep the example notebooks as ipynb files instead of converting them to python files (see below), so decided this approach would not be best for what we want in PROCESS
• nbmake doesn't seem to allow the writing of modular tests and so we did not want to use this
• testbook has an API that most aligns with our current tests, and so this is the tool that we have decided to use for testing the PROCESS example notebooks

We want to keep the examples as notebooks because, whilst running notebooks requires more packages to be installed on first running PROCESS and also debugging errors that occur in notebooks is harder, there are the advantages that running notebooks in vscode is easier than the python script version and also PROCESS docs don't render the examples yet so storing the notebooks with output is ideal

[alex-treebeard]

hi @clmould thanks for considering nbmake (I develop it).

How have you found testbook after choosing it? I'm quite open to feature requests if a pythonic interface could add something to the community here!