Decouple gallery from main documentation

[dnerini]

Currently the pysteps example gallery is part of the main pysteps repo, meaning that the static example pages are compiled by readthedocs for each change.

This slows down considerably the rtd builds, and puts us very much at the limit of their free resources.

One solution could be to fully decouple the example gallery (as a new repo, eg "pysteps-gallery") that uses GitHub actions to build and publish the example pages (although I'm not sure exactly if and how this could be possible yet).

The main advantages being that now the examples can be build only when is really necessary (eg a new release of the main package), the rtd builds wil take a fraction of the time it currently takes, and the examples will possibly have more visibility thanks to a dedicated repo.

Another interesting feature could be to use jupyter notebooks for all examples. We could then use nbsphinx to build the html pages, while also giving the possibility to open them directly with Google Colab.

[aperezhortal]

I really like the idea of separating the examples from the main repo. Specially, also migrating the examples to jupyter notebooks 🚀

One solution could be to fully decouple the example gallery (as a new repo, eg "pysteps-gallery") that uses GitHub actions to build and publish the example pages (although I'm not sure exactly if and how this could be possible yet).

We had a similar discussion a few years back here, pointing out that the workflow like the wradlib one was an interesting option to explore in the future. The future has arrived 😅
Back then, our example gallery was small enough that generating the examples in RTD was a very good solution. However, now separating the examples from the main repo seems the way to go.

One solution could be to fully decouple the example gallery (as a new repo, eg "pysteps-gallery") that uses GitHub actions to build and publish the example pages (although I'm not sure exactly if and how this could be possible yet).

Another interesting workflow is used in the astropy project, separating the examples from the main repo. These long examples (tutorials written in jupyter notebooks) are kept in this repository https://github.com/astropy/astropy-tutorials. In astropy, they use GitHub Actions to build the documentation from the jupyter notebooks, and then push the documentation to the gh-pages branch. The github repo is then configured to serve the webpage in that branch.

I will check their workflow to see how they actually build the documentation.

[aperezhortal]

Another interesting workflow is used in the astropy project, separating the examples from the main repo. These long examples (tutorials written in jupyter notebooks) are kept in this repository https://github.com/astropy/astropy-tutorials. In astropy, they use GitHub Actions to build the documentation from the jupyter notebooks, and then push the documentation to the gh-pages branch.

I just created a minimal working example of the workflow mentioned above. Here is the generated documentation https://aperezhortal.github.io/nb_gallery_template/index.html.

It seems that it won't require too much work to separate the examples from the main documentation. The only limitation of the latter approach is that only the latest version is served online.

[kmuehlbauer]

The only limitation of the latter approach is that only the latest version is served online.

If you go one step further and create additional release branches in the notebook repo, containing the rendered notebooks (via actions) you can just clone those branches during rtd build to compile final docs. That way you have the notebooks aligned with the pysteps versions.

That's the way how wradlib is doing it.

[dnerini]

Super nice @aperezhortal , thanks a lot for putting together this proof of concept! I like it A LOT! :-)

Indeed, it would be nice to have the option to navigate previous versions, as it is already possible now. We should have a closer look to how wradlib solves this. Thanks for the suggestion, @kmuehlbauer !

[kmuehlbauer]

We should have a closer look to how wradlib solves this.

I would gladly assist if you decide to go that way.

[aperezhortal]

Thanks @kmuehlbauer for the suggestion! I'll give it a try and see how it goes.

[aperezhortal]

Better late than never 😅

I followed @kmuehlbauer suggestions of separating the example gallery from the main project.
The proof of concept is implemented in the nb_gallery_template repository that mimics a project with separate example gallery.
This is essentially a minimal implementation of the wradlib workflow.

When a commit is pushed to the main branch, an empty commit is pushed to the main branch in the example gallery.
That commit triggers the actions to execute and save the notebooks in the latest branch. That action also triggers the RTD build that pulls the executed notebooks from the latest branch.
Note that the notebooks' execution is triggered by an empty commit. This is not strictly needed, however by doing so we can keep track of the commit id that triggered the entire workflow.

Here is the generated documentation https://nb-gallery-template.readthedocs.io/en/latest/.

There are 3 possible workflows that can be triggered from the main repository (pysteps in our case):

• A workflow triggered by changes to the main branch.
• A workflow triggered by changes to any other branch that triggers the GH actions.
• A workflow triggered by tagged commits in the main branch.

The readme file includes schematics of the supported workflows.

[kmuehlbauer]

@aperezhortal I've only swept quickly through this, it looks amazing! Seems I'll have to copy one or the other neat feature from here. Cool stuff!

[dnerini]

Brilliant, thanks a lot @aperezhortal, I really look forward to use your template! You think we can start the migration? I'll create a new pysteps-gallery repo soon and try with one example.

[aperezhortal]

You think we can start the migration?

Sure. I will start working on the PR updating the workflows (for v1.6) and updating the pysteps docs.