Central Istiod manages remote data plane

Please use this instruction instead.

Follow the steps below to experiment Central Istiod manage a remote data plane on a different Kubernetes cluster.

0. Configure 2 clusters to share the same trust domain, e.g. https://istio.io/docs/setup/install/multicluster/shared/#certificate-authority

1. main cluster, generate the manifest using the following yaml (https://istio.io/docs/setup/install/multicluster/shared/#main-cluster, along with centralIstiod=true)

Examples:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    gateways:
      istio-ingressgateway:
        meshExpansionPorts:
          - port: 15017
            targetPort: 15017
            name: tcp-webhook
          - port: 15012
            targetPort: 15012
            name: tcp-istiod

    global:
      multiCluster:
        clusterName: main0
      network: network1
      centralIstiod: true
      caAddress: istiod.istio-system.svc:15012

      # Mesh network configuration. This is optional and may be omitted if
      # all clusters are on the same network.
      meshNetworks:
        network1:
          endpoints:
          # Always use Kubernetes as the registry name for the main cluster in the mesh network configuration
          - fromRegistry: main0
          gateways:
          - registry_service_name: istio-ingressgateway.istio-system.svc.cluster.local
            port: 443

        network2:
          endpoints:
          - fromRegistry: remote0
          gateways:
          - registry_service_name: istio-ingressgateway.istio-system.svc.cluster.local
            port: 443

      # Use the existing istio-ingressgateway.
      meshExpansion:
        enabled: true

If you know your ingress host or IP (for testing purpose), add the following env var to the istiod deployment:

        - name: ISTIOD_CUSTOM_HOST
          value: $MAIN_INGRESS_ADDR

If you don't know yet, you may patch the deployment after you have the value.

kubectl -n istio-system --context=${MAIN_CLUSTER_CTX} patch deployment istiod --patch "
spec:
  template:
    spec:
      containers:
      - name: discovery
        env:
        - name: ISTIOD_CUSTOM_HOST
          value: ${MAIN_INGRESS_ADDR}
"

If you don't need webhook to work, okay to skip the ISTIOD_CUSTOM_HOST customization. Note: the community is working on exposing this dynamically via https://github.com/istio/api/pull/1414

2. remote cluster, deploy using the following yaml:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: true
    pilot:
      enabled: false  
    istiodRemote:
      enabled: true 
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true 
  
  addonComponents:
    prometheus:
      enabled: false
  values:
    global:
      # The remote cluster's name and network name must match the values specified in the
      # mesh network configuration of the main cluster.
      multiCluster:
        clusterName: remote0 
      network: network2        
      centralIstiod: true
      
      # Replace ISTIOD_REMOTE_EP with the the value of ISTIOD_REMOTE_EP set earlier.
      remotePilotAddress: $MAIN_INGRESS_ADDR
      caAddress: istiod.istio-system.svc:15012

    istiodRemote:
      injectionURL: https://$MAIN_INGRESS_ADDR:15017/inject

    base:
      validationURL: https://$MAIN_INGRESS_ADDR:15017/validate

3. run create secret cmd (e.g. istioctl x create-remote-secret) to create token for accessing remote cluster, from the main cluster.

Above steps deploy istiod in cluster main which can manage remote data planes in the 2nd cluster. Optional: If you need to configure cross cluster routing, deploy cluster aware gateway on both clusters.

Validate validation webhook works:

$ k apply -f istio/galley/testdatasets/validation/dataset/networking-v1beta-Gateway-invalid.yaml 
Error from server: error when creating "../istio/galley/testdatasets/validation/dataset/networking-v1beta-Gateway-invalid.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: gateway must have at least one server