feat: change Makefile to generate dedicated python requirements.txt files

[Nischay-Pro]

Description

This PR updates the Makefile to create dedicated python version folders for each requirement.

Currently, the *.in files are generic and are used as a baseline to generate custom *.txt requirement files for every Python version we currently support.

Currently, we are targeting 3.8, 3.9, 3.10, and 3.11.

Related Items

Related to #748

---

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[nhoening]

Very cool!

I believe we also should use the fitting .txt requirements (for the Python version used there) in Dockerfile, line 15.

In the end, there seems to be no difference currently in the actually-installed versions, see below (just Python3.8 requiring a few more). How were these .txt files generated? With make freeze-deps or make upgrade-deps?

± diff requirements/3.11/app.txt requirements/3.8/app.txt 
2c2
< # This file is autogenerated by pip-compile with Python 3.11
---
> # This file is autogenerated by pip-compile with Python 3.8
12a13,14
> async-timeout==4.0.2
>     # via redis
16a19,20
> backports-zoneinfo==0.2.1
>     # via workalendar
119a124,125
>     #   alembic
>     #   flask
120a127,131
> importlib-resources==6.0.0
>     # via
>     #   alembic
>     #   jsonschema
>     #   matplotlib
210a222,223
> pkgutil-resolve-name==1.3.10
>     # via jsonschema
250a264
>     #   babel
320a335
>     #   altair
349c364,366
<     # via importlib-metadata
---
>     # via
>     #   importlib-metadata
>     #   importlib-resources

[Nischay-Pro]

For that I just ran make install-for-dev to create the dep text files.

[nhoening]

Looking good.

Can you also update the Dockerfile like I asked in the last review?

[nhoening]

Does this PR depend on #771 ?

[Nischay-Pro]

Does this PR depend on #771 ?

Yes. We need to bump up the pip-tools version for it to work on Python 3.11

Else we will get getting the following error.

ImportError: cannot import name 'BAR_TYPES' from 'pip._internal.cli.progress_bars' (/opt/miniconda3/envs/flexmeasures-3.11-test/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py)
make[1]: *** [Makefile:82: freeze-deps] Error 1
make[1]: Leaving directory '/home/nischay/Git/flexmeasures'
make: *** [Makefile:41: install-for-dev] Error 2

[nhoening]

Then we should work on merging that other PR first.

[Nischay-Pro]

We cannot merge this PR yet. The tests for Python 3.8 do not pass.

I am getting the following error when trying to run tests

/opt/miniconda3/envs/flexmeasures-3.8-test/lib/python3.8/site-packages/_pytest/config/__init__.py:642: in _importconftest
    mod = import_path(conftestpath, mode=importmode, root=rootpath)
/opt/miniconda3/envs/flexmeasures-3.8-test/lib/python3.8/site-packages/_pytest/pathlib.py:565: in import_path
    importlib.import_module(module_name)
/opt/miniconda3/envs/flexmeasures-3.8-test/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
/opt/miniconda3/envs/flexmeasures-3.8-test/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
flexmeasures/api/dev/tests/conftest.py:3: in <module>
    from flexmeasures.api.v3_0.tests.conftest import add_incineration_line
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
/opt/miniconda3/envs/flexmeasures-3.8-test/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
flexmeasures/api/v3_0/tests/conftest.py:15: in <module>
    ) -> dict[str, Sensor]:
E   TypeError: 'type' object is not subscriptable
collected 0 items / 1 error

[nhoening]

@Flix6x here is another case where type annotations give problems for 3.8. Adding from __future__ import annotations at the top would solve this right?

Great that we now explicitly test what works. This PR helps with that.

[Nischay-Pro]

All tests pass for Python versioned 3.8 to 3.11

[nhoening]

Great. I want to merge this soon and are if the GitHub Action pipeline will work as we want.
But it would be smart to first release 0.15.

And for that I want you to make another small PR with the two future imports and nothing else. I would then merge that one right away.

This is to make sure that 0.15 is compatible with Python 3.8, and that we could easily leave this PR out of the release if we need to.

[nhoening]

Ah sorry, I thought the GitHub pipeline with a matrix is done here as well, but that is a follow-up PR.

Could you still make the small 3.8 compatibility PR? It's cleaner.

For merging this, I only ask that we do a quick build of the Docker image.

[nhoening]

I can confirm this branch fixes the currently broken building of the Docker image (due to the pint dependency).