brupop API server TLS cert is untrusted

[jackgill]

As I mentioned in #478, the brupop API server on one of my EKS clusters apparently has an untrusted TLS cert:

> kubectl -n brupop-bottlerocket-aws get bottlerocketshadows
Error from server: conversion webhook for brupop.bottlerocket.aws/v1, Kind=BottlerocketShadow failed: Post "https://brupop-apiserver.brupop-bottlerocket-aws.svc:443/crdconvert?timeout=30s": x509: certificate signed by unknown authority

I installed brupop using the 1.1.0 manifest file and it is working fine on several other EKS clusters deployed using the same method.
Image I'm using:
1.1.0
Issue or Feature Request:
Looking at the PKI for brupop I see a self-signed issuer cert, but I'm not clear on how this cert is supposed to be trusted. Any advice on how to troubleshoot this issue would be appreciated.

[jackgill]

I found some additional information: when describing the API server pods, I noticed this event:
MountVolume.SetUp failed for volume "bottlerocket-tls-keys" : secret "brupop-apiserver-certificate" not found
I did however see the secret so I suspect a race condition where the secret did not exist when the pod was first created, perhaps due to cert-manager running slow or something. What's confusing is that I recreated all the pods, the event did not recur, and yet I am still seeing the same error when trying to list bottlerocketshadows.

[gthao313]

@jackgill Can you share with us how you install brupops? like installing cert-manager first and then install brupop? I'm trying to reproduce this issue.

Usually we need cert-manager running on the EKS cluster first and then install brupop after few minus when we confirm cert-manager is running.

[gthao313]

@jackgill I just reproduce the same error MountVolume.SetUp failed for volume "bottlerocket-tls-keys" : secret "brupop-apiserver-certificate" not found by installing brupop immediately after running cert-manager installation. I think maybe "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s" the cert-manager endpoint just not ready.

But this maybe not same as your situation.

[jackgill]

@gthao313 cert-manager was installed via helm long before brupop was deployed on the cluster. Usually we use terraform to apply the brupop manifests but I've noticed that it creates the resources out of order, which I thought might be causing this problem. So I tried installing brupop with kubectl apply -f but I still saw this issue.

[gthao313]

Thanks for the info. Can you check if secret brupop-apiserver-certificate is on the cluster under brupop-bottlerocket-aws namespace. If the secret isn't here, we can target on it might be something wrong on the brupop deployment. Otherwise, it should be TLS cert issue.

[jackgill]

The secret is there currently. I think that it might not have been there when the pods were first created, so I tried recreating the pods now that the secret exists. However the same error remains.

[stefan-lipinski]

We were facing the same problem today. In cert-manager logs:

cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "root-certificate\" not found" "certificate"={"Namespace":"brupop-bottlerocket-aws","Name":"root-certificate

In my point of view in bottlerocket-update-operator.yaml line 15

cert-manager.io/inject-ca-from: brupop-bottlerocket-aws/root-certificate

needs to be replaced by

cert-manager.io/inject-ca-from: brupop-bottlerocket-aws/brupop-selfsigned-ca

[jackgill]

Update: I am able to reproduce this on version 1.3.0 of brupop, installed via Helm. However, based on @stefan-lipinski's comment above, I deployed the CRD using a manifest file which I had edited to specify brupop-selfsigned-ca instead of root-certificate and this resolved the issue.

The fix may be as simple as updating the certificate name here:

bottlerocket-update-operator/models/src/constants.rs

Line 42 in c7bc227

pub const ROOT_CERTIFICATE_NAME: &str = "root-certificate";

However I'm not familiar enough with this codebase to determine if there are any other impacts to making that change.