Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Podman is not able to create containers with additional image store. #22753

Open
mahendra77024 opened this issue May 20, 2024 · 6 comments
Open
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@mahendra77024
Copy link

mahendra77024 commented May 20, 2024

Issue Description

Hi ,

I have been trying to set up shared location for storing container images. So that, all users on system can leverage these images to create containers rather than pulling container images again. I have followed below blog and able to list the images but while creating containers it's failed with below error.

$podman run registry.fedoraproject.org/fedora ls -latr /
Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

https://www.redhat.com/sysadmin/image-stores-podman

Steps to reproduce the issue

#podman --root /var/lib/containers/test-storage pull fedora
#chmod -R a+rx /var/lib/containers/test-storage
switch to rootless user

create a storage.conf in ~/.config/containers/storage.conf

[storage]
driver = "overlay"

[storage.options]
additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE R/O
registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true
registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$podman run registry.fedoraproject.org/fedora ls -latr /
Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you received

$podman run registry.fedoraproject.org/fedora ls -latr /
Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you expected

when run podman run registry.fedoraproject.org/fedora ls -latr / , it should be able to create with container without any errors as image is available in shared location with readonly mode.

podman info output

$podman info
host:
  arch: amd64
  buildahVersion: 1.31.5
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38'
  cpuUtilization:
    idlePercent: 91.79
    systemPercent: 2.31
    userPercent: 5.9
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.9"
  eventLogger: file
  freeLocks: 2045
  hostname: xyz.example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 669
      size: 1
    uidmap:
    - container_id: 0
      host_id: 85617
      size: 1
  kernel: 4.18.0-513.24.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1786728448
  memTotal: 8059088896
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.20.12
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /tmp/podman-run-85617/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 1945104384
  swapTotal: 2147479552
  uptime: 312h 5m 19.00s (Approximately 13.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/users/xx/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/containers/test-storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.12
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
  graphRoot: /home/users/xx/.local/share/containers/storage
  graphRootAllocated: 17169383424
  graphRootUsed: 221696000
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-85617/containers
  transientStore: false
  volumePath: /home/users/xx/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1713524958
  BuiltTime: Fri Apr 19 04:09:18 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE R/O
registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true
registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)
$ podman version
Client: Podman Engine
Version: 4.6.1
API Version: 4.6.1
Go Version: go1.20.12
Built: Fri Apr 19 04:09:18 2024
OS/Arch: linux/amd64
$ cat ~/.config/containers/storage.conf
[storage]
driver = "overlay"

[storage.options]
additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

$podman info
host:
arch: amd64
buildahVersion: 1.31.5
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38'
cpuUtilization:
idlePercent: 91.79
systemPercent: 2.31
userPercent: 5.9
cpus: 2
databaseBackend: boltdb
distribution:
distribution: '"rhel"'
version: "8.9"
eventLogger: file
freeLocks: 2045
hostname: xyz.example.com
idMappings:
gidmap:
- container_id: 0
host_id: 669
size: 1
uidmap:
- container_id: 0
host_id: 85617
size: 1
kernel: 4.18.0-513.24.1.el8_9.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 1786728448
memTotal: 8059088896
networkBackend: cni
networkBackendInfo:
backend: cni
dns:
package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/libexec/cni/dnsname
version: |-
CNI dnsname plugin
version: 1.3.1
commit: unknown
package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/libexec/cni
ociRuntime:
name: runc
package: Unknown
path: /usr/bin/runc
version: |-
runc version 1.1.12
spec: 1.0.2-dev
go: go1.20.12
libseccomp: 2.5.2
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
path: /tmp/podman-run-85617/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64
version: |-
slirp4netns version 1.2.1
commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 1945104384
swapTotal: 2147479552
uptime: 312h 5m 19.00s (Approximately 13.00 days)
plugins:
authorization: null
log:

  • k8s-file
  • none
  • passthrough
  • journald
    network:
  • bridge
  • macvlan
  • ipvlan
    volume:
  • local
    registries:
    search:
  • registry.access.redhat.com
  • registry.redhat.io
  • docker.io
    store:
    configFile: /home/users/xx/.config/containers/storage.conf
    containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
    graphDriverName: overlay
    graphOptions:
    overlay.imagestore: /var/lib/containers/test-storage
    overlay.mount_program:
    Executable: /usr/bin/fuse-overlayfs
    Package: fuse-overlayfs-1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64
    Version: |-
    fusermount3 version: 3.3.0
    fuse-overlayfs: version 1.12
    FUSE library version 3.3.0
    using FUSE kernel interface version 7.26
    graphRoot: /home/users/xx/.local/share/containers/storage
    graphRootAllocated: 17169383424
    graphRootUsed: 221696000
    graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
    imageCopyTmpDir: /var/tmp
    imageStore:
    number: 2
    runRoot: /tmp/containers-user-85617/containers
    transientStore: false
    volumePath: /home/users/xx/.local/share/containers/storage/volumes
    version:
    APIVersion: 4.6.1
    Built: 1713524958
    BuiltTime: Fri Apr 19 04:09:18 2024
    GitCommit: ""
    GoVersion: go1.20.12
    Os: linux
    OsArch: linux/amd64
    Version: 4.6.1

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

@mahendra77024 mahendra77024 added the kind/bug Categorizes issue or PR as related to a bug. label May 20, 2024
@giuseppe
Copy link
Member

giuseppe commented May 20, 2024

how have you pulled the images in the shared store? Have you used fuse-overlays and set force_mask = "shared"?

@mahendra77024
Copy link
Author

Hi @giuseppe ,

I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work

@giuseppe
Copy link
Member

have you configured fuse-overlayfs in the storage.conf file?

@mahendra77024
Copy link
Author

Yes I did configure. Please refer postman info and steps to reproduce section for more information. Thanks

@mahendra77024
Copy link
Author

@giuseppe any help on this would be appreciated

@snowhanse
Copy link

Seems related to discussion here: #19827 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants