ngrok API Update isn't triggered when secrets change

[alex-bezek]

What happened

Some of the route-module CRD's now reference k8s secrets by name. If a secret is updated though, it won't pickup the change automatically. Since right now, we only synchronize the data when the sync function is triggered, which happens when ingress objects change, this doesn't happen until either another change happens, or the default multi-hour sync period triggers.

What you think should happen instead

When the secret is updated, we should quickly realize and update the resource in the ngrok api. 2 approaches come to mind:

• we could continually sync in a background process Synchronize ngrok resources in background process #149
• we could also watch secrets in the edges controller and do reconciliation loops when those update.

How to reproduce

Create a EndpointWebhookVerification endpoint with a secret, confirm it works, update the secret and see the ngrok api isn't updated until something triggers a reconciliation.

[alex-bezek]

One problem with watching secrets from the https edges controller is that could watch lots of unnecessary secrets and cause a lot of unnecessary calls to the ngrok api since the edge controller is not batched via the store. An alternative approach would be for the ingress controller to:

• in the driver/store when its building the desired edge objects
• for each moduleset on an edge, find all reference secrets
• take all the secrets metadata.resourceVersion fields, combine and hash them, and put an annotation on the httpsedge crd
• so if the secrets don't change, the hash remains the same. If the secrets change, the version id in k8s for the secret will update, resulting in a new hash, a new annotation, and a trigger of the reconciliation loop for the https edge controller