Polling khstate from K8s API-server every 5sec for khcheck leading to client side throttling and high latency

[sid8489]

Currently kuberhealthy-operator creates a resource called khstate for every khcheck. It keeps on polling khstate for each khcheck from k8s API server every 5 sec Here. It leads to a lot of requests to K8s API server.

We are seeing a lot of ### Waited for Xs due to client-side throttling, not priority and fairness. message in kuberhealthy-operator pods .

Also We are seeing a lot of latency in request served by kuberhealthy server (P95 ~ 53 sec for 2xx, only 50% request are served under 1sec). These metrics have been generated using standard metrics exposed by istio sidecar

Note: <50 checks are running in cluster. All above snapshots are for master pod
@integrii What's the motvation behind storing khstate as crd instead in inmemory (maybe as a concurrent hash map)?

[integrii]

This makes sense, and repeatedly polling the API server is not optimal. It seems like Kuberhealthy needs a refactor to a reflector cache for khstate and khcheck resources. Then, when a khstate is needed, Kuberhealthy will not even make a call to the API server.

The reason that khstate exists is to provide a persistent store of state that all Kuberhealthy pods can agree on. This makes Kuberhealthy highly available without complex clustering algorithms. By default Kuberhealthy comes with two pods in a deployment. The pod that comes alphabetically first is always the master pod, but either Kuberhealthy pod can service requests for status because they both just return the data from the khstate resources.

Implementing a reflector would stop the API spam, improve the speed of Kuberhealthy, and prevent the issue you are seeing here.

[integrii]

I just did some work on this... I am refactoring this to use the existing cache for khstate resources... Progress is in the khstate-cache branch.

[sid8489]

I just did some work on this... I am refactoring this to use the existing cache for khstate resources... Progress is in the khstate-cache branch.

@integrii When can we expect a release for this? Also I would like to contribute to this change in case anything is pending

[jplouis]

Trying to understand the code base. Could the StateReflector be passed into the external.Check either instead of or along with the khstatechlient? Looking at this line. That way the ext.podHasReportedInAfterTime could use the reflector cache?

[jplouis]

@integrii I did a quick and dirty pass of using the state reflector store to alleviate some API requests. It improved the throughput a little bit. Don't have the numbers handy.

master...jplouis:kuberhealthy:master

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment on the issue or this will be closed in 15 days.