Synchronize ngrok resources in background process

[alex-bezek]

Description

We created this store concept that houses all the gathered resource information, and then a Driver has a sync function which calculates the total differences, syncs those to the ngrok api, and updates the k8s api with statuses.

We started initially by just plugging this into our current reconcile loop, however we quickly found that the number of reconcile events getting re-triggered would cause sync to be called many times in a row and hit rate limits with the ngrok api. To fix this temporarily, we added a thread safe reentrance lock https://github.com/ngrok/kubernetes-ingress-controller/blob/main/internal/store/driver.go#L123-L134

Instead of doing this, we can just trigger this sync operation on some interval in a background process. This will have the bonus effect of continually reconverging any changes made directly in the ngrok account back to what matches k8s.

Use Case

a simpler sync loop to think about and continual drift correction

Related issues

No response

[alex-bezek]

This will have the bonus effect of continually reconverging any changes made directly in the ngrok account back to what matches k8s.

This is not actually true. While the store sync may run in a loop, it just ensures the CRDs in the k8s api are consistent with the calculated values from the store. Those CRD controllers would actually have to continually reconcile in a loop to achieve any drift detection and reconciliation in the ngrok api.

[alex-bezek]

One small issue that this would solve is that right now while we watch our CRDs for changes and push them into the store via https://github.com/ngrok/kubernetes-ingress-controller/blob/main/internal/store/updatestorehandler.go it doesn't trigger a re-sync. syncing in the background would automatically get CRD updates eventually, or we could just call sync in all these functions.