Add extras mechanism for finer-grained dependency selection

[marcofavoritobi]

Proposed changes

Inspired by: #36 (comment).

This commit includes the extra-dependencies mechanism of setuptools to overcome limitations specific to certain dependencies (e.g. no support for some Python interpreter versions).

The changes use the following conventions for extras names:

• [all]: install all dependencies from all extras
• [X-sampler]: install all dependencies to make X sampler to work
• [X-loss]: install all dependencies to make X loss function work.

We do not have yet an example for the last item for the moment; but for the "forward-compatibility" of the nomenclature, we leave the -sampler suffix.

E.g. for GPy, we could have the extra called gp-sampler, that installs GPy on-demand, and not installed if not needed by the user.

This commit also includes a mechanism to handle import errors for the non-installed dependencies for some components. Such a mechanism provides a useful message to the user, e.g. it raises an exception with a useful error message pointing out the missing extra in its local installation of black-it.

Fixes

Partially addresses #36

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

n/a

Further comments

This PR is mainly open for discussion, rather than a definitive contribution.

[codecov-commenter]

Codecov Report

Merging #45 (2b2f64a) into main (8afc486) will decrease coverage by 0.01%.
Report is 2 commits behind head on main.
The diff coverage is 91.83%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #45      +/-   ##
==========================================
- Coverage   91.36%   91.35%   -0.01%     
==========================================
  Files          42       43       +1     
  Lines        1737     1782      +45     
==========================================
+ Hits         1587     1628      +41     
- Misses        150      154       +4     

Files Changed	Coverage Δ
black_it/samplers/xgboost.py	96.96% <81.81%> (-3.04%)	⬇️
black_it/samplers/gaussian_process.py	96.34% <84.61%> (-2.27%)	⬇️
black_it/_load_dependency.py	100.00% <100.00%> (ø)

[marcofavoritobi]

Merge the following PRs before this one: #46, #43

[muxator]

Concept ACK, however I was not able to install the project at this PR's revision.

Python 3.10, poetry v1.5.1:

$ poetry env use python3.10
$ poetry shell
$ poetry install
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing codecov (2.1.12): Failed
  RuntimeError
   ...
  • Installing gpy (1.10.0 f63ed48): Failed
  AssertionError
   ...

For python 3.11, instead pyproject.toml needs to be updated to menion compatibility with python == 3.11.

I see you did this in feat/python3.11-bis. However, in that branch poetry.lock needs to be regenerated. Trying to regenerate it via `poetry install fails with:

Resolving dependencies... (94.7s)

Because black-it depends on GPy (^1.10.0) which doesn't match any versions, version solving failed.

I'd say we can start rebasing the PR on top of the current tip and move from there.

[muxator]

I did not remember if this PR was written before poetry had support for optional groups.

I then checked, and the "extras" syntax used here appears - indeed - to be the right one (from https://python-poetry.org/docs/master/managing-dependencies/):

Dependency groups other than the implicit main group, must only contain dependencies you need in your development process. Installing them is only possible by using Poetry.

To declare a set of dependencies, which add additional functionality to the project during runtime, use extras instead. Extras can be installed by the end user using pip.

[marcofavoritobi]

yes, we want to rely on the setuptools' extras mechanism, which is poetry-agnostic and just works with pip.

This paradigm has been used by other successful projects, e.g. OpenAI Gym: https://github.com/openai/gym#installation

[marcofavoritobi]

I added a couple of tests (24bc95e and 72829de) to verify that the dependencies checks actually work.

The commit 7e51add just skips a test on Windows due to flaky behaviour (see https://github.com/bancaditalia/black-it/actions/runs/5952400728/job/16144300857?pr=45#step:6:2581).

[marcofavoritobi]

I also added a test in 9dcc3a8 that tries to install all the extras.

This test is quite slow, hence I marked as "not_on_ci" test (i.e. skipped on CI workflows, see also 9e0e72b).

[marcofavoritobi]

Given that the change has also been documented in the README (see c740c3c#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R54-R68), I think the PR is in a good state to be reviewed and then eventually merged.