cannot run image based on sha (specific to local unpushed images) #1802

deitch · 2023-01-04T14:35:29Z

Description

I have standard lima with nerdctl, default image. I load in a simple image:

$ nerdctl image ls
REPOSITORY    TAG    IMAGE ID        CREATED               PLATFORM       SIZE       BLOB SIZE
abc           foo    28eec4daefa7    About a minute ago    linux/arm64    1.8 MiB    940.9 KiB

I try to run the image using the name:tag, that works:

$ nerdctl run --rm -it abc:foo /bin/sh
#

It also works using the canonical name:

$ nerdctl run --rm -it docker.io/library/abc:foo /bin/sh
#

But it fails if I use the sha, whether with or without the tag:

$ nerdctl run --rm docker.io/library/abc:foo@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836  /bin/sh
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving      |--------------------------------------|
elapsed: 1.1 s                                                                                 total:   0.0 B (0.0 B/s)
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving      |--------------------------------------|
elapsed: 2.2 s                                                                                 total:   0.0 B (0.0 B/s)
INFO[0002] trying next host                              error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
FATA[0002] failed to resolve reference "docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

$ nerdctl run --rm docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836  /bin/sh
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving      |--------------------------------------|
elapsed: 1.5 s                                                                                 total:   0.0 B (0.0 B/s)
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving      |--------------------------------------|
elapsed: 2.0 s                                                                                 total:   0.0 B (0.0 B/s)
INFO[0002] trying next host                              error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
FATA[0002] failed to resolve reference "docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

The hash definitely exists in containerd and nerdctl knows it, e.g. nerdctl inspect abc:foo:

[
    {
        "Id": "sha256:78a332f4204f86ea549209154aa4c800cd26c0c4bf44dfead02bb3f7a2f8bcc9",
        "RepoTags": [
            "abc:foo"
        ],
        "RepoDigests": [
            "abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836"
        ],
        "Comment": "This is an apko single-layer image",
        "Created": "0001-01-01T00:00:00Z",
        "Author": "apko",
        "Config": {
            "AttachStdin": false,
            "Env": [
                "GOPATH=/home/build/.cache/go",
                "HOME=/home/build"
            ]
        },
        "Architecture": "arm64",
        "Os": "linux",
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:ca3f2d6ea1960d6bddd093dceef35436bbcb5e58b93f576633362e137c18ec01"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

and:

$ ctr --address /proc/3245/root/run/containerd/containerd.sock image ls
REF                       TYPE                                                 DIGEST                                                                  SIZE      PLATFORMS   LABELS
docker.io/library/abc:foo application/vnd.docker.distribution.manifest.v2+json sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836 940.9 KiB linux/arm64 -

How do I run it based on hash?

I can replicate it slightly differently if I use an image that is in a registry:

$ nerdctl pull alpine:3.17
docker.io/library/alpine:3.17:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:d3156fec8bcbc7b491a4edc271a7734dcfa186fc73282d4e120eeaaf2ce95c43:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:261da4162673b93e5c0e7700a3718d40bcc086dbf24b1ec9b54bca0b82300626:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.7 s                                                                    total:  3.1 Mi (558.9 KiB/s)

$ nerdctl image ls
REPOSITORY    TAG     IMAGE ID        CREATED           PLATFORM          SIZE       BLOB SIZE
alpine        3.17    8914eb54f968    2 seconds ago     linux/arm64/v8    7.8 MiB    3.1 MiB

$ # it finds it when referenced by tag
$ nerdctl run --rm alpine:3.17 echo hello
hello

$# if I ask for it by tag and hash, or just the hash, it goes back to registry even though the hash is there
$ nerdctl run --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 echo hello
docker.io/library/alpine@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4: resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4:                    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0:                 done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:d3156fec8bcbc7b491a4edc271a7734dcfa186fc73282d4e120eeaaf2ce95c43:                   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.6 s                                                                                    total:   0.0 B (0.0 B/s)
hello

$ # if I try it again, now it finds it
$ nerdctl run --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 echo hello
hello

$ # so by hash, it only works when the tag is _not_ there
$ nerdctl image ls
REPOSITORY    TAG       IMAGE ID        CREATED               PLATFORM          SIZE       BLOB SIZE
alpine        3.17      8914eb54f968    About a minute ago    linux/arm64/v8    7.8 MiB    3.1 MiB
alpine        <none>    8914eb54f968    About a minute ago    linux/arm64/v8    7.8 MiB    3.1 MiB

The above is not too bad, if wasteful, when dealing with published images. For images that are not published, i.e. loaded in, it is a failure.

Steps to reproduce the issue

pull a known image
try to run it by tag (or no tag) and hash
it tries to pull again from the registry, even though it is local

Describe the results you received and expected

It should recognize a local digest before trying to pull.

What version of nerdctl are you using?

$ nerdctl version
Client:
 Version:	v1.0.0
 OS/Arch:	linux/arm64
 Git commit:	c00780a1f5b905b09812722459c54936c9e070e6
 buildctl:
  Version:	v0.10.5
  GitCommit:	bc26045116045516ff2427201abd299043eaf8f7

Server:
 containerd:
  Version:	v1.6.8
  GitCommit:	9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:	1.1.4
  GitCommit:	v1.1.4-0-g5fd4c4d1

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

In lima default:

Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v1.6.8
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs stargz fuse-overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
  rootless
 Kernel Version: 5.19.0-26-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.813GiB
 Name: lima-melange-builder
 ID: 48bb5881-01b0-4ea1-87a7-69822962ae46

WARNING: AppArmor profile "nerdctl-default" is not loaded.
         Use 'sudo nerdctl apparmor load' if you prefer to use AppArmor with rootless mode.
         This warning is negligible if you do not intend to use AppArmor.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

The text was updated successfully, but these errors were encountered:

djdongjin · 2023-01-04T14:41:30Z

➜  ~ snd images
REPOSITORY    TAG       IMAGE ID        CREATED         PLATFORM       SIZE         BLOB SIZE
rabbitmq      latest    90c62dd5e5fd    11 hours ago    linux/amd64    232.4 MiB    96.9 MiB
➜  ~ snd run 90c62dd5e5fd echo hi
hi
➜  ~ snd run rabbitmq echo hi
hi
➜  ~ snd run rabbitmq:latest echo hi
hi

(snd = sudo nerdctl)

deitch · 2023-01-04T14:44:10Z

You are saying I should just give it the digest? No! It cannot be that simple! 🤦‍♂️

It does get things working (and thank you indeed for that), but should we support @sha256: format?

djdongjin · 2023-01-04T14:54:38Z

but should we support @sha256: format?

what do you mean by supporting sha256 format (e.g., REPO:TAG@sha256:xxxxxx... or sha256:xxxx....)?

It supports sha256 format, just don't need to explicitly specify the sha256: prefix.

➜  ~ snd image inspect 90c62dd5e5fd
[
    {
        "Id": "sha256:3ddcc140fe5c6dda21a48a3fcc99f120f686003106a767fc1279a5dfeac32b54",
        "RepoTags": [
            "rabbitmq:latest"
        ],
        "RepoDigests": [
            "rabbitmq@sha256:90c62dd5e5fdeda5ca57268c2ea7c7a648fed9fd73e7e545bbf36b936842ef4c"
        ],
        ....
# full sha256 id
➜  ~ snd run 90c62dd5e5fdeda5ca57268c2ea7c7a648fed9fd73e7e545bbf36b936842ef4c echo hi
hi

It's the same for other commands and for docker (when a sha256 id is needed, only the string (or a unique prefix) is needed)

➜  ~ d run 9b0d21eb242c echo hi
hi
➜  ~ d images | grep 9b0d21eb242c
golang                                                 1.19-buster       9b0d21eb242c   4 weeks ago    935MB

(d = docker)

deitch · 2023-01-04T17:04:26Z

$ docker run -it --rm alpine:3.17
/ #
$ docker run -it --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
/ #

Retrospection · 2023-02-27T13:42:37Z

I may try to solve this~

suyanhanx · 2023-02-27T13:49:20Z

I took a look and it seems that it's because it didn't match during the image walk. The tag was not matched as part of the digest, so it only matches successfully when there is only the tag.

I observed that Docker's behavior is as expected.

Please pay more attention to *ImageWalker.Walk. @Retrospection

Retrospection · 2023-02-27T14:07:21Z

I took a look and it seems that it's because it didn't match during the image walk. The tag was not matched as part of the digest, so it only matches successfully when there is only the tag.

I observed that Docker's behavior is as expected.

Please pay more attention to *ImageWalker.Walk. @Retrospection

Thanks for your kind suggestion ~

deitch added the kind/unconfirmed-bug-claim Unconfirmed bug claim label Jan 4, 2023

AkihiroSuda added bug Something isn't working and removed kind/unconfirmed-bug-claim Unconfirmed bug claim labels Jan 4, 2023

AkihiroSuda changed the title ~~cannot run image based on sha~~ cannot run image based on sha (specific to local unpushed images) Jan 4, 2023

suyanhanx mentioned this issue Mar 1, 2023

hello-world and hello-world@sha256:... are treated as different images #2061

Open

This was referenced Mar 2, 2023

[DRAFT] fix: enable run image command use sha #2068

Closed

fix: enable run image command use sha #2069

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot run image based on sha (specific to local unpushed images) #1802

cannot run image based on sha (specific to local unpushed images) #1802

deitch commented Jan 4, 2023

djdongjin commented Jan 4, 2023 •

edited

deitch commented Jan 4, 2023

djdongjin commented Jan 4, 2023

deitch commented Jan 4, 2023

Retrospection commented Feb 27, 2023

suyanhanx commented Feb 27, 2023 •

edited

Retrospection commented Feb 27, 2023 •

edited

cannot run image based on sha (specific to local unpushed images) #1802

cannot run image based on sha (specific to local unpushed images) #1802

Comments

deitch commented Jan 4, 2023

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of nerdctl are you using?

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Host information

djdongjin commented Jan 4, 2023 • edited

deitch commented Jan 4, 2023

djdongjin commented Jan 4, 2023

deitch commented Jan 4, 2023

Retrospection commented Feb 27, 2023

suyanhanx commented Feb 27, 2023 • edited

Retrospection commented Feb 27, 2023 • edited

djdongjin commented Jan 4, 2023 •

edited

suyanhanx commented Feb 27, 2023 •

edited

Retrospection commented Feb 27, 2023 •

edited