sigill on linux/arm64 (emulated)

[srenatus]

With this main.go (files here):

package main

import (
	"log"
	"time"

	"github.com/bytecodealliance/wasmtime-go"
)

func main() {
	bs, err := wasmtime.Wat2Wasm(`
	  (func (export "loop")
	    (loop br 0))
	`)
	check(err)

	for i := 0; ; i++ {
		config := wasmtime.NewConfig()
		config.SetInterruptable(true)
		store := wasmtime.NewStore(wasmtime.NewEngineWithConfig(config))

		module, err := wasmtime.NewModule(store.Engine, bs)
		check(err)

		instance, err := wasmtime.NewInstance(store, module, nil)
		check(err)

		handle, err := store.InterruptHandle()
		check(err)

		go func() {
			time.Sleep(100 * time.Millisecond)
			handle.Interrupt()
		}()
		_, err = instance.GetFunc(store, "loop").Call(store)
		if err == nil {
			panic("expected error")
		}
		if t, ok := err.(*wasmtime.Trap); ok {
			log.Printf("%d: trap: %s", i, t.Message())
		}
	}

}

func check(e error) {
	if e != nil {
		panic(e)
	}
}

The following command instantly fails for me:

$ docker run -it -v $(pwd):/src -w /src --platform linux/arm64 golang:1 go run main.go
go: downloading github.com/bytecodealliance/wasmtime-go v0.33.1
SIGILL: illegal instruction
PC=0x555a80e0b0 m=0 sigcode=2
instruction bytes: 0x50 0x76 0x93 0x0 0x55 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0

goroutine 0 [idle]:
runtime: unknown pc 0x555a80e0b0
stack: frame={sp:0x555a80ce50, fp:0x0} stack=[0x5500001bc8,0x5500800be0)

runtime: unknown pc 0x555a80e0b0
stack: frame={sp:0x555a80ce50, fp:0x0} stack=[0x5500001bc8,0x5500800be0)


goroutine 1 [syscall]:
runtime.cgocall(0x64df70, 0x4000123bf8)
        /usr/local/go/src/runtime/cgocall.go:156 +0x50 fp=0x4000123bb0 sp=0x4000123b70 pc=0x5b4bf0
github.com/bytecodealliance/wasmtime-go._Cfunc_wasmtime_func_call(0x12fb1d0, 0x400013a0c0, 0x0, 0x0, 0x0, 0x0, 0x400013a100)
        _cgo_gotypes.go:2076 +0x44 fp=0x4000123bf0 sp=0x4000123bb0 pc=0x645d94
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func1.1({0xf3d4c0, 0x4000118220}, 0x400013a0c0, 0x0, {0x12ebcc8, 0x0, 0x0}, 0x0, {0x12ebcc8, 0x0, ...}, ...)
        /go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.33.1/func.go:430 +0xbc fp=0x4000123c50 sp=0x4000123bf0 pc=0x64886c
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func1(0x400013a100)
        /go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.33.1/func.go:430 +0xa8 fp=0x4000123cd0 sp=0x4000123c50 pc=0x648788
github.com/bytecodealliance/wasmtime-go.enterWasm({0xf3d4c0, 0x4000118220}, 0x4000123e48)
        /go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.33.1/func.go:513 +0x60 fp=0x4000123d20 sp=0x4000123cd0 pc=0x648a70
github.com/bytecodealliance/wasmtime-go.(*Func).Call(0x400013a0c0, {0xf3d4c0, 0x4000118220}, {0x0, 0x0, 0x0})
        /go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.33.1/func.go:414 +0x1b8 fp=0x4000123ea0 sp=0x4000123d20 pc=0x648078
main.main()
        /src/main.go:35 +0x1c4 fp=0x4000123f70 sp=0x4000123ea0 pc=0x64d114
runtime.main()
        /usr/local/go/src/runtime/proc.go:255 +0x284 fp=0x4000123fd0 sp=0x4000123f70 pc=0x5e7a64
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000123fd0 sp=0x4000123fd0 pc=0x613564

r0      0x4000000000
r1      0x0
r2      0x4000002000
r3      0x55009379d8
r4      0x0
r5      0x17f
r6      0x2000555a7ee023
r7      0x6474e550
r8      0x84
r9      0x4
r10     0x1
r11     0x0
r12     0x12fdbe0
r13     0x55009379d8
r14     0x0
r15     0xffffffff
r16     0x1
r17     0x5500908a40
r18     0x0
r19     0x0
r20     0x555a80e580
r21     0x555a80e1c0
r22     0x0
r23     0xb24924
r24     0xffffffffffffffff
r25     0x5500800990
r26     0x1
r27     0x1
r28     0x10
r29     0x555a80e0a0
lr      0x555a80e0b0
sp      0x555a80ce50
pc      0x555a80e0b0
fault   0xffffffffbf0ed000
exit status 2

It works fine and loops for a while if run using --platform linux/amd64 or if not using docker at all, on macos/amd64 host. Since qemu and binfmt_misc is involved when running a different platform, I don't know if the problem happens on a actual linux/arm64 hardware, or if this is somehow emulation-related.

[alexcrichton]

Running the prograrm natively with go version go1.16.6 linux/arm64 it works alright for me, and in docker on the same machine it unfortunately doesn't get past this line. My guess though is that this is likely related to one of the emulation layers involved?

[akirilov-arm]

@alexcrichton What is your kernel version? For example, Ubuntu 18.04 (which, unfortunately in this case, is still supported) by default ships with a kernel that does not support the variant of the membarrier() system call you are pointing at (i.e. the kernel is older than 4.16).

[alexcrichton]

My uname -a reports as:

Linux arm2-ci.infra.bytecodealliance.org 4.19.0-9-arm64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) aarch64 GNU/Linux

So I think it may be supported? I'll note that the membarrier call only fails inside of docker for me, natively it works just fine (not sure what's happening there, I'm just sort of naively assuming it's some docker thing)

[akirilov-arm]

You are probably right - Docker stopped blocking membarrier() by default less than 2 years ago. You can try this suggestion from the documentation to unblock it.

[alexcrichton]

Ah perfect! Pasing --security-opt seccomp=unconfined to docker run I'm able to successfully run this program on Linux arm64 hardware, even in docker. That may mean @srenatus that the issue here is Docker's arm64 emulation on x86_64? Or I'm not actually sure, what is the host system you're running on when you're passing --platform linux/arm64 to docker and seeing the crash?

[srenatus]

I'm not actually sure, what is the host system you're running on when you're passing --platform linux/arm64 to docker and seeing the crash?

Host is macos 12.1 (Darwin Kernel Version 21.2.0 Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64), running docker desktop 4.1.1... docker tells me this much about its kernel:

$ docker run -it --platform linux/amd64 alpine uname -a 
Linux fe90a75cad3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 Linux
$ docker run -it --platform linux/arm64 alpine uname -a 
Linux dbab877dfae4 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 aarch64 Linux

[alexcrichton]

Ah ok, looks like you're emulating linux-arm64 on macos-x86_64. AFAIK that's a pretty serious emulation layer and given that the issue only happens there I'd probably chalk it up to the emulation layer for now.

[srenatus]

FWIW I've noticed this originally emulating arm64 on Linux/amd64, in a GitHub action run. But I've got little control over that environment, so I've taken the laptop for reproducing the error. Probably still got something to do with the emulation involved... 🤷

[srenatus]

@alexcrichton thanks for taking the time to look into this with me ❤️