Skypilot reuse hard drives

[yuval-airis]

I plan to use Skypilot on (Mainly AWS) preemptible instances (Spots). My code requires a lot of input coming through the hard drive, which I would rather not recreate on every machine creation.

Is it possible to request Skypilot to reuse given (possibly read-only) hard drives between runs/on preemptions?

[concretevitamin]

Is it possible to use a S3 bucket? It can be mounted to each SkyPilot cluster. It is natively supported in Managed Spot (https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html#checkpointing-and-recovery) either for read-only datasets or read-write checkpoints.

The mounting support works for object storage buckets. Supporting EBS is tracked in #2387.

[yuval-airis]

Unfortunately no. I didn't measure the required runtime, but I am thinking of somewhere along the lines of 500GB-1000GB data randomly accessible to many short runs (10s of seconds).

If I do that, I imagine I'll need to download all of that information and only then start my run which would increase my runtime significantly.

[concretevitamin]

Are you thinking of using EBS or EFS?

[yuval-airis]

@concretevitamin EFS

[concretevitamin]

Here's a quick gist demonstrating setting up EFS: https://gist.github.com/concretevitamin/eec319cea8b2bac3ac03f23f111c0561

Note the caveats at the top of the YAML. Also, EFS is region-specific (meaning the search space used by SkyPilot to launch compute is restricted to 1 region, reducing GPU availability).

If this proves useful, we could look into supporting it more natively (e.g., cross-region).