Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post "https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s": context deadline exceeded

[michaelst]

After installing this through helm, I can't take any action, for example restart a deployment. I get this error
error: failed to patch: Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post "https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s": context deadline exceeded

I'm using cosign.

I saw the previous issue that mentioned this error but it doesn't solve my problem, I have disabled all network policies. There is also nothing in the logs besides health and ready checks.

This is the values.yaml that I am using

deployment:
  replicasCount: 1
  podSecurityPolicy:
    enabled: true

validators:
# static validator that allows each image
- name: allow
  type: static
  approve: true
- name: default
  type: cosign
  trust_roots:
  # default is our pub key
  - name: default
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeDPf0YgntelKvfWxob8UiQscwCek
      nQnVe2HCkFLiZeA35vSgaszplMrUxw2/dkWRnWiQxVB4QjBgC+Nd3lJ91w==
      -----END PUBLIC KEY-----
  # public key securesystemsengineering repo including Connaisseur images
  # !this key is critical for Connaisseur!
  - name: securesystemsengineering-official
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
      d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
      -----END PUBLIC KEY-----
  - name: docker-official
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOXYta5TgdCwXTCnLU09W5T4M4r9f
      QQrqJuADP6U7g5r9ICgPSmZuRHP/1AYUfOQW3baveKsT969EfELKj1lfCA==
      -----END PUBLIC KEY-----

policy:
- pattern: "us-docker.pkg.dev/project/images/*:*"
- pattern: "k8s.gcr.io/*:*"
  validator: allow
- pattern: "gke.gcr.io/*:*"
  validator: allow
- pattern: "gcr.io/datadoghq/*:*"
  validator: allow
- pattern: "docker.io/istio/*:*"
  validator: allow
- pattern: "docker.io/cloudflare/cloudflared:*"
  validator: allow
- pattern: "traefik:*"
  validator: allow
- pattern: "1password/scim:*"
  validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
  with:
    key: securesystemsengineering-official
- pattern: "docker.io/library/*:*"
  with:
    key: docker-official

[xopham]

@michaelst thanks for reaching out. Would really like to get behind this.
The error indicates that the webhook timed out before Connaisseur gave a response. I suppose we should reduce the internal timeouts as in #448 , however that would only make it more explicit. I have a few questions to triangulate this:

• From your values.yaml, it seems you are using the helm native installation via the charts from artifacthub. Is that correct or is the values.yaml truncated for simplicity?
• What flavor and version of k8s do you use?
• What and how did you try to deploy (kubectl command)?
• Do you get a kubectl log message (error, warning) after 300s (though the cosign integration should not make requests by itself)?
• Is there anything in the kubectl describe of the connaisseur pods, replicaset or deployment?
• I see you activated podSecurityPolicy. Is a PSP required in your cluster?

Finally to help debugging and checking whether a request at least arrives at Connaisseur, could you set the loglevel to debug via in the values.yaml: https://github.com/sse-secure-systems/connaisseur/blob/master/helm/values.yaml#L156
Sorry, if this is all very vague poking around, but hope we find a solution.

[michaelst]

Turns out it was a firewall rule, but outside the cluster, with GKE you need to enable ports on the nodes, so I had to allow port 5000

[hhyeh]

Why does it need port 5000 for the nodes? that seems like a security vulnerability. Is there other way? I am getting the exact issue. what is port 5000 for?

[xavidop]

It worked for me!! thanks for your help! It would be fantastic if you add this to the documentation. Here my Terraform code:

resource "google_compute_firewall" "gke_connaisseur_control_plane" {
  name        = "gke-${var.cluster_name}-connaisseur-allow"
  project     = var.project_id
  description = "Allows the GKE master CIDR to communicate with the Connaisseur admission webhook."
  network     = var.vpc_network_name

  allow {
    protocol = "tcp"
    ports    = ["5000"]
  }

  source_ranges = [
    var.k8s_master_ipv4_cidr
  ]

  target_tags   = [
    "k8s-nodes"
  ]
}

[xavidop]

Here you can find a Terraform code that enables admission webhook on GKE

#481 (reply in thread)