Tutorial: Deploy Spin Operator with Flux and Kustomize

[endocrimes]

https://fluxcd.io/flux/components/kustomize/kustomizations/ - but config/default rather than kustomize as the path is probably all this needs but it'll take a bit of testing in the process of writing docs.

[bplasmeijer]

As a long flux user, I'm happy to help.

[bplasmeijer]

https://fluxcd.io/flux/components/kustomize/kustomizations/ - but config/default rather than kustomize as the path is probably all this needs but it'll take a bit of testing in the process of writing docs.

Any branch I can have look on @endocrimes ? Happy to help.

I did start from scratch now.

[kingdonb]

We are planning on doing Flux + SpinKube on the Live Code Tuesday, next week:

• https://www.youtube.com/watch?v=IXi2lhRjOlU

SpinKube and Flux automated, on Kubernetes 1.30 w/ Kingdon Barrett

I've been testing CozyStack (by Ænix) and Talos Extensions, which are both another tangent

I plan to demonstrate this using a setup similar to flux2-helm-kustomize-example, where there are HelmReleases that install CRDs, and several things that need to be deployed in different order (kwasm-operator for now, runtimeclass, tenant config, the spin app kubernetes deployment itself)

Meanwhile did anyone make progress on this issue who wants to share what they have? I finally got Flux bootstrapping on a CozyStack tenant cluster yesterday, with some help, I've already tested spin on both Talos bare metal clusters and Kamaji virtual control plane (kubevirt-based) clusters, was able to make both scenarios work.

The structure of the demo is going to be pretty simple, I think, we will bootstrap Flux into an existing repo that I set up for the demo, then take a look at what configs were loaded, the applier Kustomization and HelmReleases will all be in "suspend"

I will probably use a UI like Capacitor to unfreeze them and kick them to start reconciling, so I don't have to set up receivers

We might do a second demo later, where spin apps are published using CI/CD on GitHub Actions, the OCI repository hosting is used for the wasm publishing, and OCI Flux repo is also used for deploying the spin app.

Reach out to me on the CNCF slack before Tuesday if you want to contribute something for the demo!

[kingdonb]

re: the other tangent

I'd love to orchestrate Talos with Flux, but I haven't found a way to do that yet. Cozystack embeds Flux, but it does it in a way that I'd call "a bit un-fluxy" which I'll concede is to keep things simple at installation time. Some Helm charts get packed into a Docker image that runs its own embedded Helm Repository service in the management cluster. Flux is applied by the cozy installer, not bootstrapped, and the HelmRepository and HelmRelease artifacts representing the platform get applied directly – no GitOps. Tenant clusters of course inherit some services from the parent (root) cluster, or they can replace it with their own (etcd, ingress, monitoring). I think Tenants would bootstrap their own Flux and may never know where else Flux is used.

Since a big part of SpinKube is getting the spin shim installed, one way or another, this stuff is all relevant, even if Talos is a separate tangent, immutable kube OS is undeniably a popular idea and kwasm-operator likely won't ever work with that.

A third hypothetical demo could show how to build and deploy the containerd-shim-spin in case you find a bug, want to fix it, and test the fix. I don't know why anyone would use this stuff if they were not interested in building from source, so I think this will be interesting. If I was to rebuild the shim spin, testing it on a tenant KubeVirt+Kamaji cluster would be ideal, rather than what I find would be very impractical to test, building a full Talos OS image from scratch, plus extensions!

The complexity of building Talos Extensions (which is seemingly very high, and maybe a bit under-documented compared to installing Talos off-the-shelf) was what drove me to get Cozystack working for this demo, since virtual control plane nodes are virtual machines with their own control planes and a "regular non-Talos kubelet" on each VM, they can work like any other Linux runs Kubernetes, without requiring extensions to be built a special way in order to get some binary onto the host. You can use kwasm-operator and the privileged mode trick to get the spin shim onto the nodes hosting kubelets where you want to run spin apps.

On the other hand I think you basically need to build the entire Talos Linux OS image to build a new Talos Extension, (but maybe someone knows how to do it simpler than that)

Another option, if this stuff is interesting, is of course to tune in on Tuesday and bring your questions for the live stream!