How to "re-configure cert-manager" to use AAD Workload Identity in deployKF

[edi-bice]

The instructions here assume Helm - not sure that is the case with deployKF

https://cert-manager.io/docs/configuration/acme/dns01/azuredns/#reconfigure-cert-manager

extraManifests and cert_manager.controller ... annotations seem promising

[thesuperzapper]

@edi-bice thank you for reminding me!

To use Azure Workload Identity for cert-manager authentication, we need to add new values that let you set, ServiceAccount LABELS (not annotations) and Pod LABELS.

EDIT: after checking the Azure docs, I think there is a typo in the cert-manager AKS docs, because the required labels/annotations are:

• Pod - LABEL: azure.workload.identity/use: "true"
• ServiceAccount - ANNOTATION: azure.workload.identity/client-id: "...."

Currently, we only provide deploykf_dependencies.cert_manager.controller.serviceAccount.annotations, which sets ServiceAccount annotations. It should be relatively straightforward for us to add new values in the next release:

• deploykf_dependencies.cert_manager.controller.serviceAccount.labels
• deploykf_dependencies.cert_manager.controller.podLabels

---

However, you can use the internal helm chart values before then with the deploykf_dependencies.cert_manager.valuesOverrides value (we use the upstream cert-manager helm chart internally).

For example:

deploykf_dependencies:
  cert_manager:
    valuesOverrides: |
      cert-manager:
        podLabels:
          azure.workload.identity/use: "true"

    controller:
      serviceAccount:
        annotations:
          azure.workload.identity/client-id: "00000000-0000-0000-0000-000000000000"

          ## only needed if want a non-default AZURE_TENANT_ID
          #azure.workload.identity/tenant-id: "00000000-0000-0000-0000-000000000000"

[thesuperzapper]

@edi-bice that out-of-sync behavior is caused by a strange thing that AKS does called the Admission Enforcer. It's been discussed upstream in the cert-manager repo, and you need to label OR annotate your webhooks with admissions.enforcer/disabled: "true".

I have been digging into it more, and I really think that Microsoft needs to remove this feature from AKS, because it's just a bad idea to randomly mutate resources submitted to the cluster, because it inevitably gets into fights with other controllers or GitOps tools.

I have raised an issue with AKS about giving a global off switch for this feature, which you can promote with your Microsoft representative: Azure/AKS#4002

Microsoft did at least say they would remove one part of the admission enforcer a few weeks ago in Azure/AKS#177, but I think we need an option to fully disable it.

---

Annoyingly, because of this, to get deployKF working on AKS there are several things you will have to do (until I can make some of these changes by default).

The following apps need the annotation for actual functional reasons (not just because of the argocd diff):

• kyverno ([Bug] Webhook Controller Endless Loop kyverno/kyverno#6539)
  • annotation can be added via existing helm values
  • this one will cause a memory leak if not annotated
• kubeflow-pipelines (Cache Webhook "disabled" by default in Azure kubeflow/pipelines#5244)
  • can NOT currently be added using existing values
  • this one could also be made into a "diff only" issue if I remove the control-plane label from the kubeflow namespace

The remaining ones are "cosmetic only", and can either be solved by the annotation or telling argocd to ignore the diff:

• cert-manager / trust-manager
  • annotation can be added via existing helm values
• istio
  • annotation can be added via existing helm values
• kubeflow katib
  • can NOT currently be added using existing values
• kubeflow poddefaults
  • can NOT currently be added using existing values

I am not sure if I should just include the annotations by default in deployKF, as they should not impact other users, or if I should just expose them all as values, and show them in a "deployKF on AKS" guide, similar to the one I am writing for EKS.

---

For reference, at least for cert-manager, you could sort of fix it by adding additional value overrides like this:

deploykf_dependencies:
  cert_manager:
    valuesOverrides: |
      cert-manager:
      
        ## add this
        webhook:
          validatingWebhookConfigurationAnnotations:
            admissions.enforcer/disabled: "true"

[thesuperzapper]

I have raised a separate deployKF issue to track issues with AKS admission enforcer:

• deploykf does not work properly on AKS due to webhook "admission enforcer" #61

As this is separate from adding values for Azure Workload Identity.

[edi-bice]

I added the webhook stanza above as well as removed the control-plane label from kubeflow namespace, rollout restarted all deployments in all namespaces and did another ./scripts/sync_argocd_apps.sh yet both dkf-dep--cert-manager and dkf-dep--istio remain OutOfSync. Will add the "admissions.enforcer/disabled: true" to other webhooks and see. BTW, earlier I used this https://azure.github.io/kubeflow-aks/main/ to deploy on AKS and was able to get things up and running with just one patch applying the same annotation to the istio-sidecar-injector MutatingWebhookConfiguration. I will see if the AKS-Construction tool provides options to turn off the admission enforcer logic though it seems from your research and the myriad of tickets that it is not yet exposed/supported

[edi-bice]

I was able to deploy successfully after changing to generator mode, basically generating the yaml, checking them in to git and specifying repo and branch to reconcile

[thesuperzapper]

@edi-bice Just following up to say deployKF v0.1.4 now works properly with Azure AKS:

• Admission Enforcer:
  • deploykf does not work properly on AKS due to webhook "admission enforcer" #61 (comment)
• Cert Manager Labels:
  • feat: add pod-labels value for cert-manager controller #88

To use deployKF with AKS, you need to set the kubernetes.azure.admissionsEnforcerFix value to true:

kubernetes:
  azure:
    admissionsEnforcerFix: true