buildah from is not working for emulated architectures #5322

eyebeem-mikel · 2024-01-15T15:58:35Z

eyebeem-mikel
Jan 15, 2024

Description
When using buildah from on an x86_64 host with --arch s390x for example, the run command fails. This happens with all emulated architectures.

Steps to reproduce the issue:
Run the following in a container on an x86_64 host using --arch s390x
1.work_container=$(buildah from fedora)
2.buildah run "${work_container}" /bin/bash -c "echo hello"

Describe the results you received:

Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:7eaf774f9ce106e2c6ce771804301f91685601eacbf0c69a6ba63eb64761082b": processing tar file(Error: unknown command "/" for "buildah"

Did you mean this?
        rm

): exit status 125
-----------------------------------------------------------------------------------------------------------------
Error 1 occurred on 31

Describe the results you expected:
hello
(this is produced by the echo hello)

Output of rpm -q buildah or apt list buildah:
Using the official image from quay

quay.io/buildah/stable:v1.33.2

Output of buildah version:

quay.io/buildah/stable:v1.33.2

Output of podman version if reporting a podman build issue:

quay.io/buildah/stable:v1.33.2

Output of cat /etc/*release:

quay.io/buildah/stable:v1.33.2

Output of uname -a:

multiple emulated platforms

Output of cat /etc/containers/storage.conf:

quay.io/buildah/stable:v1.33.2

I am also attaching some scripts and a Dockerfile to easily reproduce this problem.
The host system I used was Fedora 39. The following is a list of qemu and podman rpms installed.

qemu-common-8.1.3-1.fc39.x86_64
qemu-audio-jack-8.1.3-1.fc39.x86_64
qemu-audio-pipewire-8.1.3-1.fc39.x86_64
qemu-device-display-vhost-user-gpu-8.1.3-1.fc39.x86_64
qemu-ui-opengl-8.1.3-1.fc39.x86_64
qemu-ui-spice-core-8.1.3-1.fc39.x86_64
qemu-char-spice-8.1.3-1.fc39.x86_64
qemu-ui-spice-app-8.1.3-1.fc39.x86_64
qemu-audio-spice-8.1.3-1.fc39.x86_64
qemu-device-display-qxl-8.1.3-1.fc39.x86_64
qemu-ui-egl-headless-8.1.3-1.fc39.x86_64
qemu-ui-sdl-8.1.3-1.fc39.x86_64
qemu-audio-sdl-8.1.3-1.fc39.x86_64
qemu-block-curl-8.1.3-1.fc39.x86_64
qemu-audio-alsa-8.1.3-1.fc39.x86_64
qemu-audio-dbus-8.1.3-1.fc39.x86_64
qemu-audio-oss-8.1.3-1.fc39.x86_64
qemu-audio-pa-8.1.3-1.fc39.x86_64
qemu-block-blkio-8.1.3-1.fc39.x86_64
qemu-block-dmg-8.1.3-1.fc39.x86_64
qemu-block-gluster-8.1.3-1.fc39.x86_64
qemu-block-iscsi-8.1.3-1.fc39.x86_64
qemu-block-nfs-8.1.3-1.fc39.x86_64
qemu-block-rbd-8.1.3-1.fc39.x86_64
qemu-char-baum-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-ccw-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-gl-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-pci-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-gpu-pci-gl-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-vga-8.1.3-1.fc39.x86_64
qemu-device-display-virtio-vga-gl-8.1.3-1.fc39.x86_64
qemu-device-usb-host-8.1.3-1.fc39.x86_64
qemu-device-usb-redirect-8.1.3-1.fc39.x86_64
qemu-device-usb-smartcard-8.1.3-1.fc39.x86_64
qemu-ui-curses-8.1.3-1.fc39.x86_64
qemu-pr-helper-8.1.3-1.fc39.x86_64
qemu-ui-gtk-8.1.3-1.fc39.x86_64
qemu-user-static-xtensa-8.1.3-1.fc39.x86_64
qemu-user-static-x86-8.1.3-1.fc39.x86_64
qemu-user-static-sparc-8.1.3-1.fc39.x86_64
qemu-user-static-sh4-8.1.3-1.fc39.x86_64
qemu-user-static-s390x-8.1.3-1.fc39.x86_64
qemu-user-static-riscv-8.1.3-1.fc39.x86_64
qemu-user-static-ppc-8.1.3-1.fc39.x86_64
qemu-user-static-or1k-8.1.3-1.fc39.x86_64
qemu-user-static-nios2-8.1.3-1.fc39.x86_64
qemu-user-static-mips-8.1.3-1.fc39.x86_64
qemu-user-static-microblaze-8.1.3-1.fc39.x86_64
qemu-user-static-m68k-8.1.3-1.fc39.x86_64
qemu-user-static-loongarch64-8.1.3-1.fc39.x86_64
qemu-user-static-hppa-8.1.3-1.fc39.x86_64
qemu-user-static-hexagon-8.1.3-1.fc39.x86_64
qemu-user-static-cris-8.1.3-1.fc39.x86_64
qemu-user-static-arm-8.1.3-1.fc39.x86_64
qemu-user-static-alpha-8.1.3-1.fc39.x86_64
qemu-user-static-aarch64-8.1.3-1.fc39.x86_64
qemu-block-ssh-8.1.3-1.fc39.x86_64
qemu-system-x86-core-8.1.3-1.fc39.x86_64
qemu-system-x86-8.1.3-1.fc39.x86_64
qemu-kvm-8.1.3-1.fc39.x86_64
qemu-user-static-8.1.3-1.fc39.x86_64
qemu-guest-agent-8.1.3-1.fc39.x86_64
qemu-img-8.1.3-1.fc39.x86_64
libvirt-daemon-driver-qemu-9.7.0-2.fc39.x86_64
podman-4.8.3-1.fc39.x86_64
container-selinux-2.227.0-1.fc39.noarch

eyebeem-mikel · 2024-01-15T16:00:06Z

eyebeem-mikel
Jan 15, 2024
Author

3 Files:
Dockerfile (Dockerfile with basically just the RUN command)
test.sh (test for a single platform running inside the container)
test/test_all_platforms.sh (main test program)
test_scripts.tar.gz

0 replies

rhatdan · 2024-01-15T18:40:51Z

rhatdan
Jan 15, 2024
Maintainer

Do you have the qemu-user-static package installed?

0 replies

eyebeem-mikel · 2024-01-16T07:34:02Z

eyebeem-mikel
Jan 16, 2024
Author

Hi Daniel, thanks for looking into this.
Yes qemu-user-static is installed.
Thats the rpm query on the F39 system .

rpm -qa |grep static
qemu-user-static-xtensa-8.1.3-1.fc39.x86_64
qemu-user-static-x86-8.1.3-1.fc39.x86_64
qemu-user-static-sparc-8.1.3-1.fc39.x86_64
qemu-user-static-sh4-8.1.3-1.fc39.x86_64
qemu-user-static-s390x-8.1.3-1.fc39.x86_64
qemu-user-static-riscv-8.1.3-1.fc39.x86_64
qemu-user-static-ppc-8.1.3-1.fc39.x86_64
qemu-user-static-or1k-8.1.3-1.fc39.x86_64
qemu-user-static-nios2-8.1.3-1.fc39.x86_64
qemu-user-static-mips-8.1.3-1.fc39.x86_64
qemu-user-static-microblaze-8.1.3-1.fc39.x86_64
qemu-user-static-m68k-8.1.3-1.fc39.x86_64
qemu-user-static-loongarch64-8.1.3-1.fc39.x86_64
qemu-user-static-hppa-8.1.3-1.fc39.x86_64
qemu-user-static-hexagon-8.1.3-1.fc39.x86_64
qemu-user-static-cris-8.1.3-1.fc39.x86_64
qemu-user-static-arm-8.1.3-1.fc39.x86_64
qemu-user-static-alpha-8.1.3-1.fc39.x86_64
qemu-user-static-aarch64-8.1.3-1.fc39.x86_64
qemu-user-static-8.1.3-1.fc39.x86_64

0 replies

rhatdan · 2024-01-17T21:09:42Z

rhatdan
Jan 17, 2024
Maintainer

If the user static is installed then it should work. Buildah does not do anything to effect this. When the host kernel sees an executable compiled for a different arch, the kernel is supposed to use the emulation from qemu-user-static to execute the code.

0 replies

nalind · 2024-01-17T21:48:07Z

nalind
Jan 17, 2024
Maintainer

Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:7eaf774f9ce106e2c6ce771804301f91685601eacbf0c69a6ba63eb64761082b": processing tar file(Error: unknown command "/" for "buildah" is not an error from buildah run. It looks like some kind of error from "buildah commit" or "buildah from" mixed with the error you'd get if you ran "buildah /".
I suggest you troubleshoot why the "buildah" command appears to have gotten "/" passed as its first non-option argument, and go from there.

0 replies

eyebeem-mikel · 2024-01-18T09:20:48Z

eyebeem-mikel
Jan 18, 2024
Author

Thanks Nalin for pointing this out.
I tried that manually, not using the scripts I attached in the tar file above.
This is the result:

mandres@traktor:~$ podman run --arch s390x --privileged --rm -it -v "$PWD:$PWD" --workdir "$PWD" quay.io/buildah/stable:v1.33.2-s390x bash
[root@bcf46df14b67 mandres]# buildah --arch s390x from fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob b457520870ff done   | 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:b457520870ffb00b89f9b896fb1d925c2a0b65ecc95a879fe21158fdba1bc643": processing tar file(Error: unknown command "/" for "buildah"

Did you mean this?
	rm

): exit status 125

The problem occurs already in the command buildah from fedora.
I thought maybe buildah pulls the from architecture, thats why I also tried with --arch, anyway it yields the same result.

0 replies

eyebeem-mikel · 2024-01-18T09:27:07Z

eyebeem-mikel
Jan 18, 2024
Author

I also tried to pull the correct arch by adding the tag, same result.

[root@bcf46df14b67 mandres]# buildah from fedora:39-s390x
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:39-s390x...
Getting image source signatures
Copying blob b457520870ff done   | 
Error: writing blob: adding layer with blob "sha256:b457520870ffb00b89f9b896fb1d925c2a0b65ecc95a879fe21158fdba1bc643": processing tar file(Error: unknown command "/" for "buildah"

Did you mean this?
	rm

): exit status 125
[root@bcf46df14b67 mandres]# arch
s390x

0 replies

eyebeem-mikel · 2024-01-18T10:07:22Z

eyebeem-mikel
Jan 18, 2024
Author

I ran the error case with a higher loglevel:

DEBU[0012] Applying tar in /var/lib/containers/storage/overlay/8762d4e942255ae5f22c1ca5cdcbea9118d53523b90ef15d705fc17a7574Copying blob b457520870ff done  
DEBU[0019] Error pulling candidate registry.fedoraproject.org/fedora:latest: writing blob: adding layer with blob "sha256:b457520870ffb00b89f9b896fb1d925c2a0b65ecc95a879fe21158fdba1bc643": Error processing tar file(exit status 125): unknown command "/" for "buildah"

Did you mean this?
	rm
 
Error processing tar file(exit status 125): unknown command "/" for "buildah"

Did you mean this?
	rm


adding layer with blob "sha256:b457520870ffb00b89f9b896fb1d925c2a0b65ecc95a879fe21158fdba1bc643"
github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/storage/storage_image.go:1009
github.com/containers/image/v5/storage.(*storageImageDestination).queueOrCommit
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/storage/storage_image.go:834
github.com/containers/image/v5/storage.(*storageImageDestination).PutBlobWithOptions
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/storage/storage_image.go:462
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1646
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1383
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1327
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:936
runtime.goexit
	/usr/lib/golang/src/runtime/asm_s390x.s:765
writing blob
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1651
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1383
github.com/containers/image/v5/copy.(*imageCopier).copyLayer
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:1327
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1
	/builddir/build/BUILD/buildah-1.22.0/src/github.com/containers/image/v5/copy/copy.go:936
runtime.goexit
	/usr/lib/golang/src/runtime/asm_s390x.s:765
DEBU[0019] shutting down the store

0 replies

eyebeem-mikel · 2024-01-22T08:48:20Z

eyebeem-mikel
Jan 22, 2024
Author

@rhatdan can you please take a look again? Do you have an idea why buildah from does not work on emulated platforms even with the original quay buildah image? Am I doing something wrong?

0 replies

eyebeem-mikel · 2024-01-22T13:54:07Z

eyebeem-mikel
Jan 22, 2024
Author

Just to be sure this is not a silly user error I did install a fresh KVM with fedora 39, installed podman and tried to use a Dockerfile with buildah bud, and it fails the same way. So it seems to me all usage of podman/buildah is broken when using it in an emulated container.
This is the setup shown below.
(kvm x86 fedora39 (podman run qemu s390x(buildah bud --> fails)))

0 replies

rhatdan · 2024-01-22T23:16:09Z

rhatdan
Jan 22, 2024
Maintainer

Could you give me the actual commands you are executing. The last comment above is Buildah within a Podman container?

What happens when you execute something like:
$ buildah bud --arch=s390x .

0 replies

eyebeem-mikel · 2024-01-23T12:36:38Z

eyebeem-mikel
Jan 23, 2024
Author

Could you give me the actual commands you are executing. The last comment above is Buildah within a Podman container?

podman run --arch s390x --privileged --rm -it quay.io/buildah/stable:v1.33.2-s390x buildah from fedora

What happens when you execute something like: $ buildah bud --arch=s390x .

This works. Also, the result is a usable s390x container.

@rhatdan could you please take a look and give some advice how to continue here?

0 replies

eyebeem-mikel · 2024-02-05T09:13:54Z

eyebeem-mikel
Feb 5, 2024
Author

Anybody who can shed some light on this problem?
I think this is a pretty strong restriction on using buildah for platforms where usually cross platform development is done.

0 replies

rhatdan · 2024-02-05T20:50:52Z

rhatdan
Feb 5, 2024
Maintainer

How about doing:
podman run --privileged --rm -it quay.io/buildah/stable buildah from --arch s390x fedora

0 replies

eyebeem-mikel · 2024-02-06T12:59:10Z

eyebeem-mikel
Feb 6, 2024
Author

This is starting an x86 container so that works.

podman run --privileged --rm -it quay.io/buildah/stable:v1.33.2 buildah from --arch s390x fedora
Trying to pull quay.io/buildah/stable:v1.33.2...
Getting image source signatures
Copying blob 08a20a796ba1 done   | 
Copying blob 8b4d5e79c1d0 done   | 
Copying blob 801d586e727b done   | 
Copying blob 8deae42c5ba7 done   | 
Copying blob 568bfe044e9c done   | 
Copying blob 718a00fe3212 done   | 
Copying blob 9668d30f67a0 done   | 
Copying blob 5cb71af33ace done   | 
Copying blob 996fc0f77826 done   | 
Copying config e655b3987b done   | 
Writing manifest to image destination
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob b457520870ff done   | 
Copying config 50d1b6a056 done   | 
Writing manifest to image destination
fedora-working-container

In the use case I presented I think that is quite a normal straight forward usage. On an x86 we spin off an emulated container with --arch s390x or armhf... in order to compile some binaries and then pack them in a container. The first container started with podman is the build container also used to build in OCP on Z. Thats why we use the same in the developer compile environment on an x86 machine.

So it seems the straight forward use case does not work with podman/buildah, do you agree Dan?
I suspect there is a generic nesting problem using emulated containers?!
@rhatdan

0 replies

nalind · 2024-02-06T14:12:48Z

nalind
Feb 6, 2024
Maintainer

In the use case I presented I think that is quite a normal straight forward usage. On an x86 we spin off an emulated container with --arch s390x or armhf... in order to compile some binaries and then pack them in a container. The first container started with podman is the build container also used to build in OCP on Z. Thats why we use the same in the developer compile environment on an x86 machine.

I'd recommend using the --platform flag for building for foreign architectures, and running a natively-compiled buildah binary. Base images will be pulled with preference matching the --platform setting, and an emulated compiler will usually default to producing binaries for the platform that the compiler itself was built for. It's how we build our own images.

So it seems the straight forward use case does not work with podman/buildah, do you agree Dan? I suspect there is a generic nesting problem using emulated containers?! @rhatdan

Emulation is very, very good, but it isn't perfect. Some of that is bugs, yes, but some of that is limitations of how emulation is configured, and some of it comes down to how the emulator interacts with the kernel.

0 replies

rhatdan · 2024-02-06T14:20:25Z

rhatdan
Feb 6, 2024
Maintainer

In a perfect world we would prefer to build all containers on native platforms and then assemble them into a multi-arch image. This is the goal of podman farm build, But for now this works well enough for a lot of use cases.

Emulating as little as possible is the best way forward.

0 replies

Hendrik-H · 2024-02-07T09:21:15Z

Hendrik-H
Feb 7, 2024

should this work?

podman run --privileged --rm --platform linux/s390x -it quay.io/buildah/stable:v1.33.2 buildah from fedora

Currently this fails for me in the way described above. I believe this had worked once but I can not reproduce it anymore.

2 replies

rhatdan Feb 7, 2024
Maintainer

It might work, but you are running a lot of code in emmulation there. if you do this as.
podman run --privileged --rm -it quay.io/buildah/stable:v1.33.2 buildah from --platform linux/s390x fedora

Then only the buildah container processes will run in emulation mode.

nalind Feb 7, 2024
Maintainer

This appears to be an aspect of user-space emulation that interferes with the reexec mechanism we use to have subprocesses perform tasks that would adversely affect global state if they were done in-process.

reexec sets argv[0] when it execs the subprocess to communicate to it what it should do instead of the normal buildah main() logic, and at startup, buildah checks argv[0] for one of the handful of recognized special values that indicates that it should do something else for its parent process and then exit.

When starting a process using an emulator, the kernel passes the binary as the emulator's first argument rather than the argv[0] value we set, so that the emulator can find the binary. The kernel docs mention a "P" flag, which could be used when registering an emulator, which causes the kernel to pass argv[0] to the interpreter as a second argument, presumably with any other flags that were passed on the command line following it. The interpreter has to be expecting that extra argument, though, and I don't think qemu-user does.

Except for reworking how the reexec package works, which depending on how it's done would end up requiring big changes to projects that use it, I don't see that there's a change we could make in this repository that would make the combination work.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildah from is not working for emulated architectures #5322

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 18 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

buildah from is not working for emulated architectures #5322

eyebeem-mikel Jan 15, 2024

Replies: 18 comments · 2 replies

eyebeem-mikel Jan 15, 2024 Author

rhatdan Jan 15, 2024 Maintainer

eyebeem-mikel Jan 16, 2024 Author

rhatdan Jan 17, 2024 Maintainer

nalind Jan 17, 2024 Maintainer

eyebeem-mikel Jan 18, 2024 Author

eyebeem-mikel Jan 18, 2024 Author

eyebeem-mikel Jan 18, 2024 Author

eyebeem-mikel Jan 22, 2024 Author

eyebeem-mikel Jan 22, 2024 Author

rhatdan Jan 22, 2024 Maintainer

eyebeem-mikel Jan 23, 2024 Author

eyebeem-mikel Feb 5, 2024 Author

rhatdan Feb 5, 2024 Maintainer

eyebeem-mikel Feb 6, 2024 Author

nalind Feb 6, 2024 Maintainer

rhatdan Feb 6, 2024 Maintainer

Hendrik-H Feb 7, 2024

rhatdan Feb 7, 2024 Maintainer

nalind Feb 7, 2024 Maintainer

eyebeem-mikel
Jan 15, 2024

Replies: 18 comments 2 replies

eyebeem-mikel
Jan 15, 2024
Author

rhatdan
Jan 15, 2024
Maintainer

eyebeem-mikel
Jan 16, 2024
Author

rhatdan
Jan 17, 2024
Maintainer

nalind
Jan 17, 2024
Maintainer

eyebeem-mikel
Jan 18, 2024
Author

eyebeem-mikel
Jan 18, 2024
Author

eyebeem-mikel
Jan 18, 2024
Author

eyebeem-mikel
Jan 22, 2024
Author

eyebeem-mikel
Jan 22, 2024
Author

rhatdan
Jan 22, 2024
Maintainer

eyebeem-mikel
Jan 23, 2024
Author

eyebeem-mikel
Feb 5, 2024
Author

rhatdan
Feb 5, 2024
Maintainer

eyebeem-mikel
Feb 6, 2024
Author

nalind
Feb 6, 2024
Maintainer

rhatdan
Feb 6, 2024
Maintainer

Hendrik-H
Feb 7, 2024

rhatdan Feb 7, 2024
Maintainer

nalind Feb 7, 2024
Maintainer