Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_dynamic fails on wasm32 WASI 8Core 3.x buildbot #117645

Closed
vstinner opened this issue Apr 8, 2024 · 7 comments
Closed

test_dynamic fails on wasm32 WASI 8Core 3.x buildbot #117645

vstinner opened this issue Apr 8, 2024 · 7 comments
Labels
3.13 bugs and security fixes OS-wasi

Comments

@vstinner
Copy link
Member

vstinner commented Apr 8, 2024 •
edited by bedevere-app bot

Example of recent failing build: https://buildbot.python.org/all/#/builders/1344/builds/1372

build:

0:00:42 load avg: 4.04 [ 50/473/1] test_dynamic worker non-zero exit code (Exit code 134)
test_cannot_change_globals_or_builtins_with_eval (test.test_dynamic.RebindBuiltinsTests.test_cannot_change_globals_or_builtins_with_eval) ... ok
test_cannot_change_globals_or_builtins_with_exec (test.test_dynamic.RebindBuiltinsTests.test_cannot_change_globals_or_builtins_with_exec) ... ok
test_cannot_replace_builtins_dict_between_calls (test.test_dynamic.RebindBuiltinsTests.test_cannot_replace_builtins_dict_between_calls) ... ok
test_cannot_replace_builtins_dict_while_active (test.test_dynamic.RebindBuiltinsTests.test_cannot_replace_builtins_dict_while_active) ... ok
test_eval_gives_lambda_custom_globals (test.test_dynamic.RebindBuiltinsTests.test_eval_gives_lambda_custom_globals) ... ok
test_globals_shadow_builtins (test.test_dynamic.RebindBuiltinsTests.test_globals_shadow_builtins) ... ok
test_load_global_specialization_failure_keeps_oparg (test.test_dynamic.RebindBuiltinsTests.test_load_global_specialization_failure_keeps_oparg) ... 
Error: failed to run main module `python.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x1af790 - <unknown>!compiler_visit_expr
           1: 0x1afaf9 - <unknown>!compiler_visit_expr
           2: 0x1afaf9 - <unknown>!compiler_visit_expr
           3: 0x1afaf9 - <unknown>!compiler_visit_expr
           (...)
         339: 0x1afaf9 - <unknown>!compiler_visit_expr
         340: 0x1afe9e - <unknown>!compiler_visit_expr
         341: 0x1a8f50 - <unknown>!compiler_codegen
         342: 0x1a8873 - <unknown>!_PyAST_Compile
         343: 0x20f198 - <unknown>!run_mod
         344: 0x20f8b2 - <unknown>!PyRun_StringFlags
         345: 0x1876d4 - <unknown>!builtin_eval
         346: 0xb92f8 - <unknown>!cfunction_vectorcall_FASTCALL
         347: 0x60876 - <unknown>!PyObject_Vectorcall
         348: 0x1a125a - <unknown>!_PyEval_EvalFrameDefault
         349: 0x1a1d21 - <unknown>!_PyEval_Vector
         (...)
         407: 0x233859 - <unknown>!Py_BytesMain
         408: 0x5967 - <unknown>!main
         409: 0x3ca4e3 - <unknown>!__main_void
         410: 0x5940 - <unknown>!_start
       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: memory fault at wasm address 0x1000005b0 in linear memory of size 0xa00000
    3: wasm trap: out of bounds memory access

test.pythoninfo:

_testcapi.LONG_MAX: 2147483647
_testcapi.Py_C_RECURSION_LIMIT: 500
build.NDEBUG: ignore assertions (macro defined)
build.Py_DEBUG: No (sys.gettotalrefcount() missing)
sysconfig[HOSTRUNNER]: wasmtime run --wasm max-wasm-stack=8388608 --wasi preview2 --env PYTHONPATH=/$(shell realpath --relative-to /opt/buildbot/kushaldas-wasm/3.x.kushaldas-wasi.wasi.nondebug/build/build_oot/host/../.. /opt/buildbot/kushaldas-wasm/3.x.kushaldas-wasi.wasi.nondebug/build/build_oot/host)/$(shell cat pybuilddir.txt):/Lib --dir ../..::/

sysconfig[OPT]: -DNDEBUG -g -O3 -Wall
sysconfig[PY_CFLAGS]: -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall
sysconfig[PY_CFLAGS_NODIST]: -std=c11 -Wextra -Wno-unused-parameter -Wno-int-conversion -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../../Include/internal -I../../Include/internal/mimalloc
sysconfig[PY_CORE_LDFLAGS]: -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760
sysconfig[PY_LDFLAGS_NODIST]: -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760
sysconfig[PY_STDMODULE_CFLAGS]: -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -std=c11 -Wextra -Wno-unused-parameter -Wno-int-conversion -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include

Python is built with 512 KiB stack and run with 8 MiB stack and Py_C_RECURSION_LIMIT=500.

Linked PRs
@vstinner
Copy link
Member Author

vstinner commented Apr 8, 2024

Latest successful build (4 days ago):

First failing build:

So the buildbot starts failing since the change: commit 04697bc

commit 04697bcfaf5dd34c9312f4f405083b6d33b3511f
Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Date:   Thu Apr 4 16:47:26 2024 +0100

    gh-117494: extract the Instruction Sequence data structure into a separate file (#117496)

Maybe it's just that this commit changed how many stack memory Python uses on the WASI build.

vstinner added a commit to vstinner/cpython that referenced this issue Apr 8, 2024
@vstinner
pythongh-117645: Skip test_dynamic global specialization on WASI
51a998d 
Skip test_load_global_specialization_failure_keeps_oparg() of
test_dynamic on WASI build. The test uses too much stack memory.
@bedevere-app bedevere-app bot mentioned this issue Apr 8, 2024
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Apr 8, 2024
@vstinner
pythongh-117645: Skip test_dynamic global specialization on WASI
26f9381 
Skip test_load_global_specialization_failure_keeps_oparg() of
test_dynamic on WASI build. The test uses too much stack memory.
vstinner added a commit to vstinner/cpython that referenced this issue Apr 8, 2024
@vstinner
pythongh-117645: Skip test_dynamic global specialization on WASI
5e69fe2 
Skip test_load_global_specialization_failure_keeps_oparg() of
test_dynamic on WASI build. The test uses too much stack memory.
vstinner added a commit that referenced this issue Apr 8, 2024
@vstinner
gh-117645: Skip test_dynamic global specialization on WASI (#117646)
ac45766 
Skip test_load_global_specialization_failure_keeps_oparg() of
test_dynamic on WASI build. The test uses too much stack memory.
@vstinner vstinner closed this as completed Apr 8, 2024
vstinner added a commit to vstinner/cpython that referenced this issue Apr 9, 2024
@vstinner
pythongh-117645: Increase WASI stack size from 512 KiB to 8 MiB
6187587
vstinner added a commit to vstinner/cpython that referenced this issue Apr 9, 2024
@vstinner
pythongh-117645: Increase WASI stack size from 512 KiB to 8 MiB
a190743
@bedevere-app bedevere-app bot mentioned this issue Apr 9, 2024
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Apr 9, 2024
@vstinner
pythongh-117645: Increase WASI stack size from 512 KiB to 8 MiB
2736f25 
Increase also the initial memory from 10 MiB to 20 MiB.

Reenable test_dynamic on WASI release build.
@ericsnowcurrently ericsnowcurrently added OS-wasi 3.13 bugs and security fixes labels Apr 11, 2024
vstinner added a commit that referenced this issue Apr 16, 2024
@vstinner
gh-117645: Increase WASI stack size from 512 KiB to 8 MiB (#117674)
9197847 
Increase also the initial memory from 10 MiB to 20 MiB.

Reenable test_dynamic on WASI build.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-117645: Skip test_dynamic global specialization on WASI (pyt…
29ed527 
…hon#117646)

Skip test_load_global_specialization_failure_keeps_oparg() of
test_dynamic on WASI build. The test uses too much stack memory.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-117645: Increase WASI stack size from 512 KiB to 8 MiB (pyth…
3cff98f 
…on#117674)

Increase also the initial memory from 10 MiB to 20 MiB.

Reenable test_dynamic on WASI build.
@markshannon
Copy link
Member

WASI CI jobs still seem rather fragile: https://github.com/python/cpython/actions/runs/8937187388/job/24548976147?pr=118322

@vstinner
Copy link
Member Author

vstinner commented May 3, 2024

WASI CI jobs still seem rather fragile: https://github.com/python/cpython/actions/runs/8937187388/job/24548976147?pr=118322

test_glob failed on this job. Please open a new issue. This issue was about test_dynamic.

@markshannon
Copy link
Member

test_glob or test_dynamic doesn't matter. The issue is the WASI stack depth, it has little to do with the particular tests that fail.

@vstinner
Copy link
Member Author

vstinner commented May 3, 2024

The error message is different.

       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: wasm trap: call stack exhausted

@markshannon
Copy link
Member

I suspect that is just an artifact of the emscriptem compiler and where it puts large auto variables. The root problem is the same.

@vstinner
Copy link
Member Author

vstinner commented May 3, 2024

I cannot reproduce the test_glob issue on the main branch or on the your #118322 PR. You should open an issue with more details explaining how to reprodudce the issue, if Python was built in debug or release mode, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.13 bugs and security fixes OS-wasi
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vstinner @ericsnowcurrently @markshannon