Extension registration and automation for gardener-operator

[timuthy]

How to categorize this issue?

/area usability
/area ops-productivity
/area open-source
/area ipcei
/kind enhancement

What would you like to be added:
The following concept aims at improving the overall experience and productivity for operators. A Gardener installation usually needs additional and tedious preparation tasks to be done, e.g. creating storage buckets for backups or managing DNS entries. All of those can be automated via gardener-operator.
They even overlap with requirements that were already implemented for shoot clusters, but never made it to the Garden due to conceptional reasons.

Therefore, we plan to add a new custom resource Extension:

🎯 Goals

• Registration of a provider extension for the Garden-Runtime and Shoot clusters at the same time.
  • Serves common tasks in Runtime cluster, e.g. reconciling DNSRecord or BackupBucket resources (gardener-operator will need to create those).
  • Will be translated into and deployed as ControllerRegistration/ControllerDeployment (ref) in Garden cluster.
• Registration of an optional admission controller (example).
• Stick to Helm deployment logic (ref) but only rely on OCI repositories to fetch charts/assets (similar to Flux's support for OCI Repositories).
• Add great defaulting for known extensions, hosted in github.com/gardener. This allows smaller and trial landscapes to be setup with a minimal Extension resources (see Example 2 below).

🙅‍♂️ Non Goals

• It's not planned to eliminate ControllerRegistrations. For complex or advanced configurations, operators should still consider their additional usage.

Extension API:

Fields of the Extension API are very similar to ControllerRegistration/ControllerDeployment as requirements and goals overlap for the main part.

Example 1:

apiVersion: operator.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: aws
spec:
    resources: # optional - defaulted by Operator for well-known extensions
  # - kind: BackupBucket
  #   type: aws
  #   primary: true|false
  #   globallyEnabled: true|false # only valid if kind=Extension
  #   reconcileTimeout: 30s # only valid if kind=Extension
  #   lifecycle: {}
  #   workerlessSupported: true|false # only valid if kind=Extension
  version: # optional - defaulted by Operator for well-known extensions
  deployment: # optional - defaulted by Operator for well-known extensions
    admission:
      ociRepositoryURL: # optional - defaulted by Operator for well-known extensions
      type: helm
      providerConfig:
        values: {}
    extension:
      ociRepositoryURL: # optional - defaulted by Operator for well-known extensions
      type: helm
      providerConfig:
        values: {}
    # policy: OnDemand|Always

The resources configuration merges the extension handling for Garden and Shoot clusters, whereas the most part is irrelevant for the Garden and only needed to craft the ControllerRegistration.

With decent defaulting, we want the extension registration for operators to be as simple as the following:

Example 2:

apiVersion: operator.gardener.cloud
kind: Extension
metadata:
  name: aws

[timuthy]

/cc @MartinWeindel @rfranzke @oliver-goetz @dkistner