How to use istio-csr in "multi-primary in different networks" deployment

[ku9nov]

Hello everyone,

I have successfully configured two Kubernetes clusters using Istio in different networks (without using istio-csr). I also have a successfully configured cluster in one region using istio-csr, which seems to be working as intended with HTTPS and more.

Now, I am trying to create a cluster in another region and set up multi-cluster on different networks using istio-csr. The first one works, but there is no connection between the clusters (using this: https://istio.io/latest/docs/setup/install/multicluster/verify/).

It seems that the issue lies in the fact that the root certificates created automatically on both clusters are different, as outlined in this guide: https://cert-manager.io/docs/tutorials/istio-csr/istio-csr/.

If I try to use the root certificate from the main cluster, I get errors like:

2023-12-22T12:36:18.172013Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"istio-ca\")"

If I use the certificate generated during creation, the connection between clusters doesn't work.

How can I use a single root certificate for both clusters? Or any additional ideas?

[ku9nov]

I think it's important to mention that I am using a self-signed certificate as a client.

# SelfSigned issuers are useful for creating root certificates
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: istio-system
spec:
  selfSigned: {}
---
# Request a self-signed certificate from our Issuer; this will function as our
# issuing root certificate when we pass it into a CA Issuer.

# It's generally fine to issue root certificates like this one with long lifespans;
# the certificates which istio-csr issues will be much shorter lived.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  isCA: true
  duration: 87600h # 10 years
  secretName: istio-ca
  commonName: istio-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  subject:
    organizations:
    - cluster.local
    - cert-manager
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
# Create a CA issuer using our root. This will be the Issuer which istio-csr will use.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  ca:
    secretName: istio-ca

[ku9nov]

Okay, it seems like it's fixed. The magic is in extracting the generated root certificate from the first cluster and correctly adding it to the second one without overwriting, so the second cluster considers it its own.

If implemented through Ansible, it would look like this:
issuer.j2

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: istio-system
spec:
  selfSigned: {}
---
{%- if cluster_type == 'main' %}

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  isCA: true
  duration: 87600h # 10 years
  secretName: istio-ca
  commonName: istio-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  subject:
    organizations:
    - cluster.local
    - cert-manager
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  ca:
    secretName: istio-ca
{%- endif %}

Ansible tasks:

- name: Apply Issuer manifest
  kubernetes.core.k8s:
    state: present
    template: issuer.j2

- name: Retrieve tls.crt from istio-ca secret in "{{ istio_ns_name }}" namespace
  k8s_info:
    api_version: v1
    kind: Secret
    name: istio-ca
    namespace: "{{ istio_ns_name }}"
  register: secret_info
  when: cluster_type == 'main'

- name: Decode and save tls.crt to root-ca.pem
  shell:
    cmd: echo "{{ secret_info.resources[0].data['tls.crt'] | b64decode }}" > {{ ansible_env.HOME }}/.ssh/root-ca.pem
  args:
    executable: /bin/bash
  when: secret_info.resources | length > 0
  when: cluster_type == 'main'

- name: Decode and save tls.key to root-ca.key
  shell:
    cmd: echo "{{ secret_info.resources[0].data['tls.key'] | b64decode }}" > {{ ansible_env.HOME }}/.ssh/root-ca.key
  args:
    executable: /bin/bash
  when: secret_info.resources | length > 0
  when: cluster_type == 'main'

- name: Create a generic secret istio-ca in istio-system namespace in secondary cluster
  k8s:
    name: istio-ca
    namespace: istio-system
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: istio-ca
      type: Opaque
      data:
        ca.crt: "{{ lookup('file', '{{ ansible_env.HOME }}/.ssh/root-ca.pem') | b64encode }}"
        tls.crt: "{{ lookup('file', '{{ ansible_env.HOME }}/.ssh/root-ca.pem') | b64encode }}"
        tls.key: "{{ lookup('file', '{{ ansible_env.HOME }}/.ssh/root-ca.key') | b64encode }}"
    state: present
  when: cluster_type == 'secondary'

- name: Create Issuer in secondary cluster
  k8s:
    state: present
    definition:
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        ca:
          secretName: istio-ca
  when: cluster_type == 'secondary'

Now the check for multi-primary clusters in different networks is working.

I implemented this through Ansible and haven't verified it in the terminal, but without Ansible, it would look something like this:
⚠️ This manifest should be added to both clusters!

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: istio-system
spec:
  selfSigned: {}

than:

1. Decode and save tls.crt to root-ca.pem from first cluster

echo "$(kubectl get secret istio-ca -n istio-system -o jsonpath='{.data.tls\.crt}' | base64 -d)" > ~/.ssh/root-ca.pem

2. Decode and save tls.key to root-ca.key from first cluster

echo "$(kubectl get secret istio-ca -n "{{ istio_ns_name }}" -o jsonpath='{.data.tls\.key}' | base64 -d)" > ~/.ssh/root-ca.key

3. Create a generic secret istio-ca in istio-system namespace in secondary cluster

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: istio-ca
  namespace: istio-system
type: Opaque
data:
  ca.crt: "$(cat ~/.ssh/root-ca.pem | base64)"
  tls.crt: "$(cat ~/.ssh/root-ca.pem | base64)"
  tls.key: "$(cat ~/.ssh/root-ca.key | base64)"
EOF

4. Create Issuer in secondary cluster

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  ca:
    secretName: istio-ca
EOF