Refactor packaging

[dnerini]

Hi everyone,
I don't know if the topic has already come up but I would suggest splitting the library into multiple components (eg: core, visualization, etc.) so that you can only install what you need (eg: downscaling) and avoid what you don't need (eg: eg: visualizattion) and all related dependencies (eg: matplotlib).

I say this for all those like me who find themselves installing libraries in AWS Lambda (and similar) where space is precious and therefore it is very convenient to be able to install only what you need without having to manually delete files and folders after installation.

Originally posted by @fox91 in #216 (comment)

[aperezhortal]

I like the idea that @dnerini suggested in #216, add different options to install a particular set of dependencies depending on the needs of the users using the extras_require in setup.cfg. We can implement by:

• Refactor visualization module to handle matplotlib as an optional dependency
  • To avoid the try:except blocks in each of the .py files in the visualization module, we can move the matplotlib import to the init.py in the visualization module, and import it from that module as from pysteps.visualization.matplotlib. We can also add a function like check_matplotlib_import() that raises the exception if matplotlib is not present. What do you think?
• Create different installation configurations using the extras_require option setup.cfg, one for each functionality and also bundles (like [dev], [all], etc.).

---

In #216 I also suggested splitting the library into different components using something like Namespace packages. However, if the goal is avoiding ithe installation of unnecessary optional depencies, the optional-dependencies handling in setuptools seems to me a better (and simpler) alternative.

[dnerini]

Thanks @aperezhortal for the nice summary, we should give it a try!
I also like the proposition for making matlplotlib an optional dependency, although this might need to be applied to few other modules (eg we have a plot module in pysteps.verification).

One aspect that is not clear to me is how is this going to translate in terms of conda-forge packages? Do you know?

One more thing we should discuss is a tentative list of sub-functionalities, we already mentioned a few:

• visualisation (or a shorter 'plot'): matlplotlib, (cartopy)
• dev: pytest, black

what else could be useful?

• doc: sphynx
• ...

[aperezhortal]

One aspect that is not clear to me is how is this going to translate in terms of conda-forge packages? Do you know?
Conda-forge allows different outputs. Although it never use this functionality, we can use as reference the matplotlib-feedstock. Matplotlib creates 3 outputs, matplotib-base, matplolib, and mpl_sample_data.

One more thing we should discuss is a tentative list of sub-functionalities, we already mentioned a few

What about:

• all: full functionality
• motion: all the optional dependencies in the motion module
• vis: matlplotlib
• geovis: matlplotlib, cartopy, pyproj
• dev: pytest, black
• doc: sphynx

[ladc]

I am not an expert in python packaging but I'm all for having the proposed separately installable components. For example, doc requires more than simply sphynx, it also uses e.g. a bibtex package. It would be useful to be able to install all doc requirements with one doc submodule and likewise for the rest. Perhaps something like that was already possible, but I missed it...

[fox91]

I would not want the aspect I brought to light to get confused with the difference between development dependencies and production dependencies: for development dependencies (pytest, black, sphynx, etc.) specific separations are not necessary, indeed it would be harmful.

For example, I use poetry instead of setuptools and here is the dedicated dev-dependencies section for dependencies of this type that will be installed only by those who install the library in development mode.
I can't name the specific option to get the same effect in setuptools but I think there is an equivalent.