Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many local (according to nodejs) in generated Wasm module #21847

Open
mkarup opened this issue Apr 29, 2024 · 4 comments
Open

Too many local (according to nodejs) in generated Wasm module #21847

mkarup opened this issue Apr 29, 2024 · 4 comments

Comments

@mkarup
Copy link

mkarup commented Apr 29, 2024

I have some C code generated through the CertiCoq project, specifically a verified implementation of sha256.
Emscripten produces a Wasm module that contains a function that fails when trying to run the module via the generated js scaffolding and nodejs (v.20.12.2).

I have attached a zip containing the C files needed for running the emcc command:
sha_fast.zip

Error message:

failed to asynchronously prepare wasm: CompileError: WebAssembly.instantiate(): Compiling function #141:"body" failed: local count too large @+339007
Aborted(CompileError: WebAssembly.instantiate(): Compiling function #141:"body" failed: local count too large @+339007)
/home/martin/Documents/sha_fast/sha_fast-opt_O0.js:132
      throw ex;
      ^

RuntimeError: Aborted(CompileError: WebAssembly.instantiate(): Compiling function #141:"body" failed: local count too large @+339007)
    at abort (/home/martin/Documents/sha_fast/sha_fast-opt_O0.js:593:11)
    at /home/martin/Documents/sha_fast/sha_fast-opt_O0.js:707:5

Node.js v20.12.2

I haven't tested other Wasm runtimes (so disregard this if it is a nodejs related issue).
If i add the -O1 optimization flag, the module instantiates and runs, but the behavior (printed result) differs from that of the x86-binary, similar to this issue (let me know if you need the details for this specific instance of the difference in behavior)

Emscripten Version

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.58 (a41843e0860e52c948c1fce20307933c6631c800)
clang version 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b0dbcb69d81d7c2)
Target: wasm32-unknown-emscripten
Thread model: posix

Full command:

emcc -v -O0 -o sha_fast-opt_O0.js -w -Wno-everything -I./ sha_fast_main_wasm.c gc_stack.c CertiCoq.Benchmarks.tests.sha_fast.c glue_sha_fast.c -sALLOW_MEMORY_GROWTH=1 -sSTACK_SIZE=1000000 --profiling

Output:

 /home/martin/code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/home/martin/code/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -v -O0 -w -Wno-everything -I./ sha_fast_main_wasm.c -c -o /tmp/emscripten_temp_g7hbpg60/sha_fast_main_wasm_0.o
clang version 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b0dbcb69d81d7c2)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/martin/code/emsdk/upstream/bin
 (in-process)
 "/home/martin/code/emsdk/upstream/bin/clang-19" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name sha_fast_main_wasm.c -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/home/martin/Documents/sha_fast -v -fcoverage-compilation-dir=/home/martin/Documents/sha_fast -resource-dir /home/martin/code/emsdk/upstream/lib/clang/19 -D EMSCRIPTEN -I ./ -isysroot /home/martin/code/emsdk/upstream/emscripten/cache/sysroot -internal-isystem /home/martin/code/emsdk/upstream/lib/clang/19/include -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include -O0 -Werror=implicit-function-declaration -Wno-everything -w -ferror-limit 19 -fvisibility=default -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fignore-exceptions -fcolor-diagnostics -iwithsysroot/include/fakesdl -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/emscripten_temp_g7hbpg60/sha_fast_main_wasm_0.o -x c sha_fast_main_wasm.c
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 .
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/fakesdl
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/compat
 /home/martin/code/emsdk/upstream/lib/clang/19/include
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 /home/martin/code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/home/martin/code/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -v -O0 -w -Wno-everything -I./ gc_stack.c -c -o /tmp/emscripten_temp_g7hbpg60/gc_stack_1.o
clang version 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b0dbcb69d81d7c2)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/martin/code/emsdk/upstream/bin
 (in-process)
 "/home/martin/code/emsdk/upstream/bin/clang-19" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name gc_stack.c -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/home/martin/Documents/sha_fast -v -fcoverage-compilation-dir=/home/martin/Documents/sha_fast -resource-dir /home/martin/code/emsdk/upstream/lib/clang/19 -D EMSCRIPTEN -I ./ -isysroot /home/martin/code/emsdk/upstream/emscripten/cache/sysroot -internal-isystem /home/martin/code/emsdk/upstream/lib/clang/19/include -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include -O0 -Werror=implicit-function-declaration -Wno-everything -w -ferror-limit 19 -fvisibility=default -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fignore-exceptions -fcolor-diagnostics -iwithsysroot/include/fakesdl -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/emscripten_temp_g7hbpg60/gc_stack_1.o -x c gc_stack.c
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 .
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/fakesdl
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/compat
 /home/martin/code/emsdk/upstream/lib/clang/19/include
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 /home/martin/code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/home/martin/code/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -v -O0 -w -Wno-everything -I./ CertiCoq.Benchmarks.tests.sha_fast.c -c -o /tmp/emscripten_temp_g7hbpg60/CertiCoq.Benchmarks.tests.sha_fast_2.o
clang version 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b0dbcb69d81d7c2)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/martin/code/emsdk/upstream/bin
 (in-process)
 "/home/martin/code/emsdk/upstream/bin/clang-19" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name CertiCoq.Benchmarks.tests.sha_fast.c -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/home/martin/Documents/sha_fast -v -fcoverage-compilation-dir=/home/martin/Documents/sha_fast -resource-dir /home/martin/code/emsdk/upstream/lib/clang/19 -D EMSCRIPTEN -I ./ -isysroot /home/martin/code/emsdk/upstream/emscripten/cache/sysroot -internal-isystem /home/martin/code/emsdk/upstream/lib/clang/19/include -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include -O0 -Werror=implicit-function-declaration -Wno-everything -w -ferror-limit 19 -fvisibility=default -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fignore-exceptions -fcolor-diagnostics -iwithsysroot/include/fakesdl -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/emscripten_temp_g7hbpg60/CertiCoq.Benchmarks.tests.sha_fast_2.o -x c CertiCoq.Benchmarks.tests.sha_fast.c
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 .
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/fakesdl
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/compat
 /home/martin/code/emsdk/upstream/lib/clang/19/include
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 /home/martin/code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/home/martin/code/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -v -O0 -w -Wno-everything -I./ glue_sha_fast.c -c -o /tmp/emscripten_temp_g7hbpg60/glue_sha_fast_3.o
clang version 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b0dbcb69d81d7c2)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/martin/code/emsdk/upstream/bin
 (in-process)
 "/home/martin/code/emsdk/upstream/bin/clang-19" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name glue_sha_fast.c -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/home/martin/Documents/sha_fast -v -fcoverage-compilation-dir=/home/martin/Documents/sha_fast -resource-dir /home/martin/code/emsdk/upstream/lib/clang/19 -D EMSCRIPTEN -I ./ -isysroot /home/martin/code/emsdk/upstream/emscripten/cache/sysroot -internal-isystem /home/martin/code/emsdk/upstream/lib/clang/19/include -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include -O0 -Werror=implicit-function-declaration -Wno-everything -w -ferror-limit 19 -fvisibility=default -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fignore-exceptions -fcolor-diagnostics -iwithsysroot/include/fakesdl -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/emscripten_temp_g7hbpg60/glue_sha_fast_3.o -x c glue_sha_fast.c
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 .
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/fakesdl
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include/compat
 /home/martin/code/emsdk/upstream/lib/clang/19/include
 /home/martin/code/emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 /home/martin/code/emsdk/upstream/bin/clang --version
 /home/martin/code/emsdk/upstream/bin/wasm-ld -o sha_fast-opt_O0.wasm -L/home/martin/code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten /tmp/emscripten_temp_g7hbpg60/sha_fast_main_wasm_0.o /tmp/emscripten_temp_g7hbpg60/gc_stack_1.o /tmp/emscripten_temp_g7hbpg60/CertiCoq.Benchmarks.tests.sha_fast_2.o /tmp/emscripten_temp_g7hbpg60/glue_sha_fast_3.o -lGL-getprocaddr -lal -lhtml5 -lstubs-debug -lnoexit -lc-debug -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-debug-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpj5x0rc3qlibemscripten_js_symbols.so --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=_emscripten_stack_alloc --export=__get_temp_ret --export=__set_temp_ret --export=__wasm_call_ctors --export=_emscripten_stack_restore --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --export-table -z stack-size=1000000 --max-memory=2147483648 --initial-heap=16777216 --no-entry --stack-first --table-base=1
 /home/martin/code/emsdk/upstream/bin/llvm-objcopy sha_fast-opt_O0.wasm sha_fast-opt_O0.wasm --remove-section=.debug* --remove-section=producers
 /home/martin/code/emsdk/upstream/bin/wasm-emscripten-finalize -g --dyncalls-i64 --pass-arg=legalize-js-interface-exported-helpers sha_fast-opt_O0.wasm -o sha_fast-opt_O0.wasm --detect-features
 /home/martin/code/emsdk/node/16.20.0_64bit/bin/node /home/martin/code/emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmp9r6txfje.json
@kripken
Copy link
Member

kripken commented Apr 29, 2024

"Too many locals" is common in unoptimized builds of machine-generated code - it is very easy for such code to end up with more locals than wasm VMs will allow, unfortunately. Optimizing is what fixes it.

If you see incorrect behavior in optimized builds please reduce it as much as possible and provide that. It may also be useful to use sanitizers like UBSan to make sure you are not seeing different results due to undefined behavior.

@mkarup
Copy link
Author

mkarup commented May 19, 2024

Sorry for the late reply. Adding the -O1 flag fixes the issue with too many locals, so it's only about the incorrect behavior.

If I compile with emcc -O1 -fsanitize=undefined ... I get:

glue_sha_fast.c:71:31: runtime error: load of misaligned address 0x00168714 for type 'unsigned long long', which requires 8 byte alignment
0x00168714: note: pointer points here
  00 04 00 00 0c 87 16 00  00 08 00 00 14 87 16 00  01 00 00 00 01 04 00 00  01 00 00 00 01 04 00 00

The offending function is this

unsigned long long get_boxed_ordinal(value $v)
{
  return (unsigned long long) *((unsigned long long *) $v + -1LL) & 255LL;
}

I think it is a similar issue to what I described in my latest comment in this issue, except it doesn't pretty print the correct result, even with the -O1 flag (maybe this issue should be combined with that one?)

A little background: CertiCoq translates Coq inductive values to in-memory data structures using pointers, and it seems to only happen inductive types with more than 3 arguments.

Here is a zip of a 'reduced' exampled (Still CertiCoq, which includes a bunch of extra stuff for the runtime, I'm sorry, but most of the files are probably not relevant for this issue)

Expected output is (Three (S O) (S (S O)) true), from clang -O1 -fsanitize=undefined -w -Wno-everything -I. i_main.c glue_i.c gc_stack.c CertiCoq.Benchmarks.tests.i.c, and I get no warnings about misalignment.

With emcc -O1 -fsanitize=undefined -w -Wno-everything -I. i_main.c gc_stack.c CertiCoq.Benchmarks.tests.i.c glue_i.c and running the js with node I get

glue_i.c:51:31: runtime error: load of misaligned address 0x00046d74 for type 'unsigned long long', which requires 8 byte alignment
0x00046d74: note: pointer points here
  00 04 00 00 6c 6d 04 00  00 0c 00 00 64 6d 04 00  74 6d 04 00 01 00 00 00  00 00 00 00 00 00 00 00
              ^ 

and it doesn't print anything.

@juj
Copy link
Collaborator

juj commented May 19, 2024

I wonder was there a Binaryen pass that would have split too large functions to smaller ones that would not exceed the local count?

Recently I've been struggling with some wasm builds that also generate too many locals, and was thinking it would be nice to be able to split the files up, instead of having to -O2 it (in my case -O1 wasn't oddly enough).. and in this scenario I was looking to debug the generated code in browser debugger, so the -O2 flag was a bit problematic.

@kripken
Copy link
Member

kripken commented May 20, 2024

@mkarup which clang did you test with? If you test with the same one as emcc uses (the one bundled with the emsdk) then it should be apples to apples. Otherwise there is a risk the emsdk's clang is newer and catches more undefined behavior.

I don't know enough about UB to tell if that line

  return (unsigned long long) *((unsigned long long *) $v + -1LL) & 255LL;

has UB or not.

@juj

I agree a tool for this could be useful. Splitting might not always be easy to do though, if the locals all have huge live ranges. Another option might be to define another wasm memory (thanks to multimemory) and use space there instead of locals, basically translate local operations to ones on that memory, but that would require maintaining a stack on that memory. A way without a stack could be to use Wasm GC: allocate a struct on function entry and use that (or rather a bunch of structs, because there is a limit on struct size).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants