Remove hard ArgoCD dependency

[AndersBennedsgaard]

Checks

• I have searched the existing issues.
• This request is NOT specific to the CLI. (If so, please open an issue on the CLI repo)

Motivation

ArgoCD is a fine tool deploy stuff to a Kubernetes cluster, but not everybody want to use it (they could be using a different GitOps engine such as Flux, Fleet, etc., or not wanting to use GitOps).
It should be possible to just generate raw Kubernetes manifests as output to a directory, without the ArgoCD dependency.

I think it is fine to keep a guide for using ArgoCD with deployKF, and perhaps keep it as a recommendation, but we should not do what KubeFlow has done and add arbitrary dependencies to their platform which aren't actually needed for a lot of people (in my mind, their implementation of Istio is this is the biggest contributor to KubeFlow being hard to install and manage)

Implementation

No response

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[thesuperzapper]

@AndersBennedsgaard While I agree that it would be nice to use deployKF without ArgoCD, there are a few things that deployKF uses ArgoCD for, which can't be solved when using a raw manifests directory:

1. Cleaning up resources:
   • If you just manually apply Kustomize manifests to a Kubernetes cluster, there is no easy way to remove them again once they are no longer needed.
   • Obviously, this is less important for Helm apps, but deployKF includes both Helm and Kustomize apps (notably Kubeflow, lol).
2. Syncing in a controlled way:
   • We have a reasonably complex sync process which uses ArgoCD to ensure that each app is properly synced before moving on to the next.
   • Within many apps, we make use of the argocd.argoproj.io/sync-wave feature, to prevent race conditions (e.g. to ensure cert-manager is synced and healthy before syncing ClusterIssuer resources)
   • Many of our apps use helm.sh/hook: pre-install,pre-upgrade hooks (e.g. to generate unique OIDC secrets) these take a non-trivial amount of time. Using ArgoCD allows us to only run them when their app is "out of sync" which we could not possibly know without ArgoCD.

Given the complexity of achieving all this natively in the deploykf CLI, and the significantly lower popularity of FluxCD, it's just really not a priority at the moment (and I am sure you would prefer I spend my time adding user-facing features!)
Do you have a specific thing which is driving your desire to not use ArgoCD?

---

PS: from my discussions with large companies all around the world, pretty much everyone who is using Kubernetes seriously is using ArgoCD. For this and many other reasons, ArgoCD is how I recommend anyone deploy something as complex as deployKF. Yes, it has its problems, but for better or worse, it is the standard.

[AndersBennedsgaard]

Actually, no, I wouldn't prefer that you spend time on user-facing features. I would prefer that the focus is on the stability and availability of use. But you probably prefer to focus on user-facing features as it is just more fun to work on 😉

But I have to disagree with several things you write.

If you just manually apply Kustomize manifests to a Kubernetes cluster, there is no easy way to remove them again once they are no longer needed.

While I agree that a good way of removing stuff should be possible, is is very easy to just write kubectl delete -f /path/to/stuff with a potential -R for recursively deleting stuff. I am not advocating for using raw kubectl commands (even automated commands using Jenkins or similar); I am just saying that without the raw manifests, you remove a lot of potential users from using your product (which is fine if that is the decision).

We have a reasonably complex sync process which uses ArgoCD to ensure that each app is properly synced before moving on to the next.

As I read it, this script is making sure that all the manifests are correctly applied to the cluster, in the correct order, and verifying that some resources has been pruned?
If that is the case: that is basically a solved problem in Flux. Here you just commit stuff to git, and specify which "folders"/"paths" are depending on each other. Then Flux will try to apply the contents of one path before moving on to the next.
I would be surprised if ArgoCD and other similar tools doesn't have this feature built in?

Within many apps, we make use of the argocd.argoproj.io/sync-wave feature, to prevent race conditions (e.g. to ensure cert-manager is synced and healthy before syncing ClusterIssuer resources)

Same as above

Many of our apps use helm.sh/hook: pre-install,pre-upgrade hooks (e.g. to generate unique OIDC secrets) these take a non-trivial amount of time. Using ArgoCD allows us to only run them when their app is "out of sync" which we could not possibly know without ArgoCD.

This is just a Helm feature, so ArgoCD isn't really needed for this?

Do you have a specific thing which is driving your desire to not use ArgoCD?

Yes. I've used ArgoCD, and I am not very impressed. It has a very nice developer-friendly UI, and if you don't do pure GitOps then it's a very nice alternative to kubectl. But as I am an infrastructure guy for the most part, I require stability when working with mutating webhooks, CRDs, dependencies to other stuff, and so on, which ArgoCD does not handle well. As I also want to use GitOps, I do not want to have state in both ArgoCDs database, git, and Kubernetes at the same time. I only want it to be in git, which ArgoCD can do with read-only mode, but it is janky at best.
All of this is solved by Flux, and other GitOps engines.

PS: from my discussions with large companies all around the world, pretty much everyone who is using Kubernetes seriously is using ArgoCD. For this and many other reasons, ArgoCD is how I recommend anyone deploy something as complex as deployKF. Yes, it has its problems, but for better or worse, it is the standard.

While it is probably true that ArgoCD is the most used engine (I am not entirely sure that it is, since a lot of people use them in conjunction), I've worked with a lot of different companies in my role as consultant, and I've only seen the use of ArgoCD once - when I wanted to test it on one of our demo setups.
It might be that I am working in a more security-heavy field where we require an audit-trail for all changes (mostly in the form of the git history), or something else, but as I see it (and mentioned above), choosing ArgoCD as the only way to support your tool will not make it usable for a LOT of people.
Which might be fine. In that case, feel free to close this issue.

[thesuperzapper]

@AndersBennedsgaard my comment in #110 (comment) was primarily about why raw manifests wouldn't work. However, it's always been a roadmap item to support FluxCD (given it's the only other widely used K8S CD tool).

If you or your customers would benefit from FluxCD support coming sooner, I am very happy to help facilitate such a contribution from you or your team (or others watching this thread).

[AndersBennedsgaard]

I am not actually arguing for full-blown Flux support, but rather removal of the hard ArgoCD dependency. It's fair to have support for ArgoCD, Flux, whatever other system using plugins or some other ways, but there are times where raw manifests are just preferred (such as when using Flux).
Unfortunately, I don't have the most time in the world right now, but if I do find some I would love to take a stab at it

[Zeratoxx]

Is a Kubernetes Operator an option?

Instead of relying on ArgoCD, a Kubernetes Operator is a Kubernetes Native opportunity to do automated cluster management on the fly.

[mnasty]

I would like to second this request. As a lead engineer on an ML team functioning like a startup and provisioning our own siloed infrastructure for compliance reasons, it really sucks to have to do a complete deployment of a separate product just for a single deployment of something unrelated.. especially because our cluster is used primarily for data manipulation and model development rather than service hosting in production.

I understand that this is a very complex deployment but focusing on simpler or built-in solutions to the challenges makes your product more accessible. Any external dependency creates more points of failure and maintenance.

Do the benefits really outweigh the cost for every team and every use case? What good are great features on a product nobody wants to go through the struggle of deploying?

There is elegance in simplicity.