Integration of KubeStellar and SkyPilot

[amanroa]

I have been trying to integrate KubeStellar and SkyPilot. I believe that because KubeStellar can support multiple K8s clusters while SP can only support one - at this moment - adding KubeStellar to SkyPilot would be a cool idea!

This is how far I've gotten in doing an integration, and here's my high level architecture diagram.

If anyone has any comments as to how I can complete this integration, or comments on the process in general, I would greatly appreciate it! And if anyone is interested in replicating this, I have instructions here. Thanks!

[amanroa]

Will reopen at a later time with additional details!

[clubanderson]

@waltforme @clubanderson

[concretevitamin]

cc @tonxxd who's looked into multi-cluster in a k8s setting, and @romilbhardwaj

[clubanderson]

@romilbhardwaj and @tonxxd - anything you can suggest to help us get over this hurdle?

[romilbhardwaj]

This is an interesting project - thanks for sharing!

I'm not familiar with Kubestellar, but if it exposes the same Kubernetes API as any other cluster, then it should work with Skypilot by putting your kubeconfig at ~/kube/config and opening ports 30000-32767 on your nodes. You can find more documentation here.

Let me know if you run into any issues or have any questions.

[clubanderson]

@romilbhardwaj thank you for your comment. If you watch the video you will see the issue. We did supply the kubeconfig. We believe something else is going on within the skypilot framework.

[tonxxd]

Thanks @concretevitamin
We are looking into skypilot+kubernetes but i think from a different perspective than the one described here.
Our current working prototype uses skypilot at the kubernetes scheduler level to distribute workloads outside the cluster on ephemeral GPU vms. We are doing this to maintain compatibility with existing kubernetes frameworks and without code changes while lowering costs by having GPUs scheduled only when needed.

[romilbhardwaj]

@clubanderson From the video it seems like sky launch completed successfully (3:52 in the video). Is there some behavior occurring that you were not expecting? Perhaps I'm missing something.

[waltforme]

@amanroa You showed us that, sky launch returns some errors when you ran it against a KubeStellar workload description space (WDS). Could you expose more details on what are the returned errors so that it can be better discussed here?

[clubanderson]

Folks, we have to regroup on this. Looks like we have a bit more work to do. We will reopen this issue in the future when we have some bandwidth to work together. Thank you.