health-checks: zero route availability improvements

[wasaga]

Summary

This PR does some improvements to the Zero Route Availability health check:

• ignores certificate errors, as the point of that health check is not to check the validity of the cert, but rather whether there is an expected pomerium instance available at a given hostname
• if check fails, re-check it with exponential back-off
• previously it was only ran on schedule (every 30 minutes), now it would also be triggered on configuration change
• also adds an "internal error" reason for health checks when health check cannot complete because of some external condition unrelated to the health check itself

Related issues

Fixes: https://github.com/pomerium/pomerium-zero/issues/2360

User Explanation

Checklist

• reference any related issues
• updated docs
• updated unit tests
• updated UPGRADING.md
• add appropriate tag (improvement / bug / etc)
• ready for review

[coveralls]

coverage: 56.37% (-0.1%) from 56.512%
when pulling 7d8af09 on wasaga/route-availability-check-improvement
into 568e99f on main.

[kenjenkins]

Why should the health checks sync config separately from the rest of the app?

What are the difference between this approach compared with registering an OnConfigChange() listener on the main config source?

[wasaga]

that's a fair point, however here we only have access to the lowest layer of config.
by design higher layers are notified when lower layer configs are changed but not the other way around.

we would have to move the checker loop initialization to an entirely different place to be able to attach to a regular config.Source; there are benefits doing that; I can make a separate PR for that.