Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using logging causes failures in some tests, including some in All post-commit tests #8371

Open
bbradelTT opened this issue May 10, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@bbradelTT
Copy link
Contributor

Describe the bug

Running at least one test with logging enabled causes failures. Specifically tests/ttnn/unit_tests/test_tracer.py::test_ttnn_bert, which is in All post-commit tests.

To Reproduce
Steps to reproduce the behavior:

  1. Set the log variables:
export TT_METAL_LOGGER_LEVEL=DEBUG
export TT_METAL_LOGGER_TYPES=Op
export TTNN_ENABLE_LOGGING=True
export TT_METAL_DPRINT_CORES=all  
  1. run
pytest "tests/ttnn/unit_tests/test_tracer.py::test_ttnn_bert[bert=models.demos.bert.tt.ttnn_optimized_bert-sequence_size=384-batch_size=8-model_name=phiyodr/bert-large-finetuned-squad2-device_l1_small_size=0]"

which is part of All post-commit tests > fd-unit-tests > ttnn group 1 grayskull E150
causes

Fatal Python error: Segmentation fault

Thread 0x00007f050d6a8740 (most recent call first):
  File "/proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/decorators.py", line 411 in call_wrapper
  File "/proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/tracer.py", line 168 in call_wrapper
  File "/proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/decorators.py", line 626 in call_wrapper
  File "/proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/decorators.py", line 701 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/models/demos/bert/tt/ttnn_optimized_bert.py", line 52 in bert_attention
  File "/proj_sw/user_dev/bbradel/tt-metal/models/demos/bert/tt/ttnn_optimized_bert.py", line 165 in bert_layer
  File "/proj_sw/user_dev/bbradel/tt-metal/models/demos/bert/tt/ttnn_optimized_bert.py", line 229 in bert
  File "/proj_sw/user_dev/bbradel/tt-metal/models/demos/bert/tt/ttnn_optimized_bert.py", line 251 in bert_for_question_answering
  File "/proj_sw/user_dev/bbradel/tt-metal/tests/ttnn/unit_tests/test_tracer.py", line 118 in test_ttnn_bert
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/proj_sw/user_dev/bbradel/tt-metal/python_env/bin/pytest", line 8 in <module>
Segmentation fault (core dumped)

Expected behavior
The test succeeds with no segmentation fault.

Please complete the following environment information:

  • OS: Ubuntu 20.04
  • Version of software: main
@bbradelTT bbradelTT added the bug Something isn't working label May 10, 2024
@arakhmati
Copy link
Contributor

This is what I see:

terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_ASSERT @ tt_metal/impl/debug/dprint_server.cpp:671: rpos <= bufsize
backtrace:
 --- /home/ubuntu/work/tt-metal/build/lib/libtt_metal.so(+0x3393c1) [0x7fffb55033c1]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe6793) [0x7ffff4665793]
 --- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7ffff7dae609]
 --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7ffff7ee8353]

@arakhmati
Copy link
Contributor

Test passes after running unset TT_METAL_DPRINT_CORES

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants