crictl v1.28.0 reporting incorrect image digest #1366
Comments
bmelbourne
added
kind/bug
|
Here is the output from |
|
@saschagrunert @haircommander is this related to the image id work? |
|
@kannon92 @bmelbourne this is likely an issue with containerd, since you can reproduce it only with You probably want to move this issue to the containerd repo. /cc @mikebrow |
|
@saschagrunert /cc @mikebrow |
|
@bmelbourne but the underlying runtime is containerd with a CRI middle layer, right? Means that containerd or the CRI shim needs to take care of the result, not crictl. |
|
@saschagrunert not sure I agree, everything provided in the description has been generated from the same If you look closely at the following output of the |
|
We get the images from the ListImages RPC and then normalize the digests by just using the first one: Lines 626 to 630 in 17b4dd6 Maybe we should take multiple digests into account, but what's the value of |
|
Thanks for the specific code snippet, I'll compare this with how Maybe something in recent |
|
Hello. In your example for crictl images you are looking at the image ID not the image ref.... At the moment (the last 6y), yes the image id we are showing on image pull / list etc. for use by the cri apis, for example to say remove the cri image.., is the config digest.. But that is just an id to reference our internal cri meta for the actual cri image. confusing if you think it's the digest of the image/index.. What you want, I believe, is repo digest: FYI cool tool that Derek put together: |
|
I get the same results with dockerd, so I don't think that it is related to containerd. It has another intesting quirk, in that filtering the images removes the digests... $ docker images --digests busybox:latest
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
busybox latest <none> 3f57d9401f8d 6 weeks ago 4.26MB
$ docker images --digests | grep busybox | grep latest
busybox latest sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74 3f57d9401f8d 6 weeks ago 4.26MBThis behaviour also spreads to cri-dockerd, so filtered images end up with $ sudo crictl images --digests busybox:latest
busybox latest <none> 3f57d9401f8d4 4.26MB
$ sudo crictl images --digests | grep busybox | grep latest
busybox latest 6d9ac9237a84a 3f57d9401f8d4 4.26MBBut otherwise, it seems to be about the "Id": "sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741",
"RepoTags": [
"busybox:latest"
],
"RepoDigests": [
"busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74"
],
|
|
Images that are built locally only have an Id, they won't get a digest until they are pushed to a repo. "RepoTags": [
"myimage:latest"
],
"RepoDigests": [], |
|
I've created a clean standalone AWS node with no pre-existing running Kubernetes cluster, and I believe the issue relates to how the Based on the following output, I'd appreciate an expert review to confirm my assumption before I close this ticket, and raise the issue with the You'll notice from the above output, |
|
You say image digest, but reference the image id? |
|
Yes, I didn't quite explain it properly, the |
|
Okay, that behaviour I can reproduce. Using |
What happened:
When attempting to use a vulnerability scanner, such as
kubevulnas part of the Kubescape security platform, thekube-apiserver v1.28.4pod is not reporting the correct image digest and hence, the image cannot be pulled fromregistry.k8s.io, scanned and verified.crictl v1.28.0
nerdctl v1.7.1
kube-apiserver v1.28.4
What you expected to happen:
Closer examination shows that
crictlis incorrectly setting the image ID in the container runtime usingconfig-sha256and notindex-sha256which matches the image digest stored in the remote kubernetesregistry.k8s.ioimage registry, and hence, security scanners such askubevulnwould be able to use the correct digest and pull the image successfully.How to reproduce it (as minimally and precisely as possible):
kubeadmand pull the kubernetesv1.28.4system imagesWhat happenedsection to review thekube-apiserverimage digestAnything else we need to know?:
Environment:
cat /etc/os-release):uname -a):The text was updated successfully, but these errors were encountered: