What is the order of precedence in case of conflicting ClusterProfiles?

[rbjorklin]

Say I have a base set of components I want applied to all my clusters:

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: base-components
spec:
  clusterSelector: env=dev
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://open-policy-agent.github.io/gatekeeper/charts
    repositoryName: gatekeeper
    chartName: gatekeeper/gatekeeper
    chartVersion:  3.9.0
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    helmChartAction: Install
  - chartName: kyverno/kyverno
    chartVersion: v3.0.1
    helmChartAction: Install
    releaseName: kyverno-latest
    releaseNamespace: kyverno
    repositoryName: kyverno
    repositoryURL: https://kyverno.github.io/kyverno/

and now I want to upgrade one of the components in this set so I create an override:

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: gatekeeper-override
spec:
  clusterSelector: cluster=dev-1
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://open-policy-agent.github.io/gatekeeper/charts
    repositoryName: gatekeeper
    chartName: gatekeeper/gatekeeper
    chartVersion:  3.10.0
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    helmChartAction: Install

Is there an order of precedence here? If so how is it determined?

EDIT: There is a chance I can solve this with: #532

[gianlucam76]

In your example, the second ClusterProfile gatekeeper-override won't be deployed.

An Helm chart (in a given managed cluster) can only be managed by a ClusterProfile. So you will see in the ClusterSummary corresponding to your second ClusterProfile a status like:

  helmReleaseSummaries:
  - conflictMessage: ClusterSummary base-components-capi-clusterapi-workload managing
      it
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    status: Conflict

In general, the order is this:

1. all helm charts listed in a ClusterProfile are deployed in the very same order. So looking at your first ClusterProfile, gatekeeper will be deployed before Kyverno (this is covered with more details here

2. A ClusterProfile might depend on another ClusterProfile. Let's say you have two ClusterProfiles (kyverno and prometheus). And you want to install Kyverno before prometheus is deployed

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile 
metadata: 
  name: prometheus
spec:
  dependsOn:
  - kyverno  # name of the ClusterProfile that must be deployed before this is deployed

This is covered in more details here

[gianlucam76]

and if you want to upgrade gatekeeper, you can simply change the version in your first clusterProfile. Though this will upgrade all clusters in your set.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: base-components
spec:
  clusterSelector: env=dev
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://open-policy-agent.github.io/gatekeeper/charts
    repositoryName: gatekeeper
    chartName: gatekeeper/gatekeeper
    chartVersion:  3.16.0 # changing the version will instruct Sveltos to upgrade (or downgrade)
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    helmChartAction: Install
  - chartName: kyverno/kyverno
    chartVersion: v3.0.1
    helmChartAction: Install
    releaseName: kyverno-latest
    releaseNamespace: kyverno
    repositoryName: kyverno
    repositoryURL: https://kyverno.github.io/kyverno/

If you want to update only one cluster to see all works just fine, you can use the maxUpdate field of the ClusterProfile along with ValidateHealths. What Sveltos will do in that case, it will trigger the upgrade only on one cluster and only when all is successful will move to next cluster

 maxUpdate: 1

This is covered here

At the moment there is no way to say this ClusterProfile takes it over for this helm charts. The easiest would probably be what you mentioned (to express the Kustomize Path as a template so different paths are used for different clusters matching same ClusterProfile).

[gianlucam76]

Though a little too manual for my taste, this is something that will work.

Set clusterProfile 1 StopMatchingBehavior to LeavePolicies
This will instruct Sveltos to not withdraw the helm charts when a cluster stops being a match.

Then change the labels of your second cluster to only match your second ClusterProfile.

At this point, your cluster will stop matching first ClusterProfile (base-components) and it will start matching the second one (gatekeeper-override)

Because StopMatchingBehavior is set to LeavePolicies on ClusterProfile base-components, gatekeeper and kyverno helm charts will be left.
the cluster will start matching ClusterProfile gatekeeper-override and given no other ClusterProfiles are managing Gatekeeper, Sveltos will upgrade it.

I know this is not exactly your use case (and as I mentioned above is also too manual for me) but just though of it and so sharing it here.

Thank you

[gianlucam76]

@rbjorklin this PR will better handle this scenario

[rbjorklin]

Thank you for implementing this! I’ll try it out as soon as I get a chance!

[gianlucam76]

Release with tiers is now out: https://github.com/orgs/projectsveltos/discussions/555

Documentation is here