-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zero values from CPU usageNanoCores stats #9531
Comments
fyi @kiashok |
@knabben Have you seen this with previous releases of 1.7.* as well or has this started happening only with 1.7.6? |
@kiashok I went from 1.6 to 1.7.6 directly |
What version of 1.6 were you on? |
UPDATE: Tried on 1.6.26 and it's still passing, the value is consistent with .5 limit and never returns 0 on CPU used tasks. Confirming it's broken on 1.7.0+ |
That's curious, from 1.6.26, the Using the command: On 1.6.26 running crictl stats directly returns
On - 1.7.0, the getUsageNanoCores was introduced, returning the value.
|
The error is on getUsageNanoCores and newUsageNanoCores variable calculation; it's expected that For this test, I have printed the Kubelet/containerd (?) is calling the CRI API by default every 10 seconds and the reason of this inconsistent behavior in the tests.
-- The first loop of curling
The second loop running in parallel
|
This 0 return behavior in the The difference between Linux and Windows in Kubelet is that Linux uses cAdvisor to overwrite the value from the CRI for final stats/summary serialization, while Windows forwards the value directly. |
@fuweid this seems to occur on Linux as well, could we add that label as well?
fyi @bobbypage @haircommander - This doesn't seem to affect Linux today since kubelet replaces the value with cAdvisor value but might be more of an issue once cri-only metrics moves to beta? |
@knabben and I looked into this a bit more and found that since this is calculated value and the value is calculated from last time it stored, and it gets store each time, when two calls come in back to back we get 0 (as would be expected since the value hasn't changed between the two calls as they were run at essentially the same time, this partly due to the granularity of the value recorded as well) So while I don't think this is necessarily a bug it does lead to an inconsistent value that is being returned for
|
Since this is related to come changes from k/k I've also written up explanation from kubelet perspective with a few options to solve it. Please comment on kubernetes/kubernetes#122092 (comment) to avoid splitting conversation |
Description
Upstream
kubelet
test that checks/stats/summary
endpoint is flaky, returning a 0 value forusageNanoCores
in a non deterministic way.kubernetes/kubernetes#122092
Steps to reproduce the issue
crictl stats -o json
in loop and observe the fluctuation of values, bringing usageNanoCores=0 in a few requestsDescribe the results you received and expected
From RPC call:
What version of containerd are you using?
containerd 1.7.0
Any other relevant information
Windows 2022 - with Kubernetes v1.23.7
Show configuration if it is related to CRI plugin.
No response
The text was updated successfully, but these errors were encountered: