[FEATURE] Waxctl deploy aws - More customizable deployments

[iusztinpaul]

I have been trying to deploy a streaming pipeline that uses huggingface transformers to embed documents.

I wanted to use waxctl deploy aws to do that, as, in theory, it sounded excellent.

Unfortunately, I had to drop using waxctl deploy aws because it had some crucial limitations that stopped me from going forward. The biggest one was not controlling how the Python requirements were installed. For example, I wanted to install torch+cpu from the torch CPU index, but somehow, during the installation, it ignored the provided index and installed the GPU version. This was a crucial step for my personal purpose.

Thus, I said to propose what I lacked during my process as potential improvements and see if you think they are a good fit for your dev plan:

1. Allowing specifying the Python version as a parameter.
2. Allowing specifying the Volume size at the beginning
3. Bring a mechanism to upload/download big files to the EC2 instance before running the dataflow.
4. Allowing running a docker image in the EC2 (more flexibility in installing and running things).

Still, I would love to see how waxctl is evolving and use it in the future, as I think it is a great idea & tool.

Never tried your K8s features, I should start from there.

Have a great day ✌🏻