[REQUEST]: Making Library support for Python happen

[partouf]

Python library support is a special beast. Python requires dependency resolution where the environment only contains 1 version. We'll need to rely on an external resolver to do the work for us.

This also results in that we'll have to accept that environments need to be built on the fly and cached. Of course if we have a limited choice in libraries, we can pre-cache a common set of combination of libraries.

Some choices to make:

• What packages and versions to allow, what are the requirements
• and if we want to allow a free-for-all - how would we do that in UI, API and sanitization
• What would be an acceptable delay in building a new environment, acceptable queue length
• and how do we show that to the user
• Which things to take into consideration for later addition of other languages and libraries. If for example we want to later add JS packages, what is important to take into account.
• Configuration choice: what format? environment.yml? requirements.txt?
• Configuration choice: allow user customization or do we generate?
• Storage choice for environment packages: S3? EFS? Conan?

Steps to take:

• Add a request queue in AWS
• Websocket to announce recently built environments so users know when they can do a compile with the environment
• Add API's that the frontend can approach safely to add requests and look into the result queue (or would some things go into the backend?)
• Setup a server that can process requests to make a new environment based on python version and environment configuration, package up the environment, store it somewhere, make it known to the websocket
• UI stuff
• APIs

[mattgodbolt]

Some thoughts:

• what packages and versions
  • one thing would be to "just" run pip and/or conda and let it work that out. basically support everything?
• free-for-all
• good point though
• showing to user...likewise. a UI would be nice but also a "requirements.txt" is more python-y (and opens doors to other stuff?). requirements.txt is pip; environmen.yml is conda, so ...both? if we support both (big lift)
• stoage -> my vote is S3. we know where we stand with that and it's cheaper. hash the "source" and then use that as the name to look for (and write to) in S3