Isolate job's requirements

[iszulcdeepsense]

A job runs a single Python interpreter process that contains 2 codebases:

• wrapper code (handling http, metrics, swagerUI, etc)
• actual job's logic

Each of them have its third-party dependencies installed with pip install -r requirements.txt.
That raises a risk of conflicting packages.
Although they are installed in 2 different virtual environments, they're still running in the same single Python process that makes it impossible to load 2 different versions of the same package at the same time.
For instance, if wrapper code relies on fastapi==0.100.0, but the job depends on fastapi==0.90.0, there might be the clash, cause the wrapper's library takes precedence when loading modules in Python and the job will use fastapi==0.90.0

I see the only good way out of here: Wrapper code shouldn't have any third-party dependency (that could be overwritten or conflicted by a job). Then, there won't be any conflict. It can either implement the necessary logic on its own or use "forked" versions of libraries (eg. FastAPI renamed to fastapi_racetrack package).

This task has a low priority as it rarely happens to break something (but when it does, it's astonishing and hard to find).