Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

segfault when running the tests #2867

Open
mgeplf opened this issue May 2, 2024 · 4 comments
Open

segfault when running the tests #2867

mgeplf opened this issue May 2, 2024 · 4 comments
Labels
bug

Comments

@mgeplf
Copy link
Collaborator

mgeplf commented May 2, 2024

Context

Overview of the issue

I saw the pytest pin, and was wondering if I could get it to happen with the older version of pytest

Expected result/behavior

Not have a segfault

NEURON setup

Commit 8a9bada; python 3.11 from pyenv on Ubuntu 20.04.

(fresh virtualenv)
$ cmake -DNRN_ENABLE_CORENEURON=ON -DNRN_ENABLE_PYTHON=ON -DNRN_ENABLE_TESTS=ON ..
$ pip install -r ../nrn_requirements.txt
$ pip install -r ../external/nmodl/requirements.txt
$ ninja
$ ctest --verbose -R pytest_coreneuron::basic_tests_py3.11
# use the output of the command to rerun tests:
$ cmake "-E" "env" [...] "python3" "-m" "pytest" "--capture=tee-sys" "./test/pytest_coreneuron"

If I run it a few times, along with failures, I get a SEGFAULT.

In gdb, I get the following traceback:

test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py test 0
error temp.tmp line 1: could not parse:

# One-point soma
#n,type,x,y,z,radius,parent
1  1 0 0 0 10 -1
2  3 0 10 0 1 1
3  3 0 20 0 1 2
4  3 0 -10 0 2 1
5  3 0 -20 0 2 4
6  3 10 0 0 .5 1
7  3 10 5 0 1 6

Thread 1 received signal SIGSEGV, Segmentation fault.
neuron::container::handle_base<neuron::container::owning_identifier<neuron::container::Node::storage> >::get_handle<neuron::container::Node::field::Voltage> (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/view_utils.hpp:93
93              return underlying_storage().template get_handle<Tag>(this->id());
(gdb) bt
#0  neuron::container::handle_base<neuron::container::owning_identifier<neuron::container::Node::storage> >::get_handle<neuron::container::Node::field::Voltage> (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/view_utils.hpp:93
#1  neuron::container::Node::handle_interface<neuron::container::owning_identifier<neuron::container::Node::storage> >::v_handle (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/node.hpp:187
#2  Node::v_handle (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/nrnoc/section.h:153
#3  mk_ttd () at /home/gevaert/tmp/nrn/src/nrniv/partrans.cpp:511
#4  0x00007f47a4780d27 in nrn_thread_memblist_setup () at /home/gevaert/tmp/nrn/src/nrnoc/multicore.cpp:647
#5  0x00007f47a47776dd in v_setup_vectors () at /home/gevaert/tmp/nrn/src/nrnoc/treeset.cpp:1697
#6  0x00007f47a476e913 in nrnhoc_topology () at /home/gevaert/tmp/nrn/src/nrnoc/solve.cpp:296
#7  0x00007f47a478a0e7 in hoc_call () at /home/gevaert/tmp/nrn/src/oc/code.cpp:1418
#8  0x00007f47a4816f1a in fcall (vself=vself@entry=0x7f47a4ccdb90, vargs=vargs@entry=0x7f47a6d8bb18 <_PyRuntime+58904>) at /home/gevaert/tmp/nrn/src/nrnpython/nrnpy_hoc.cpp:728
#9  0x00007f47a4710ace in OcJump::fpycall (f=f@entry=0x7f47a4816dd0 <fcall(void*, void*)>, a=a@entry=0x7f47a4ccdb90, b=b@entry=0x7f47a6d8bb18 <_PyRuntime+58904>) at /home/gevaert/tmp/nrn/src/nrniv/ocjump.cpp:138
#10 0x00007f47a4813f0a in hocobj_call (self=0x7f47a4ccdb90, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwrds=<optimized out>) at /home/gevaert/tmp/nrn/src/nrnpython/nrnpy_hoc.cpp:796
#11 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a4ccdb90, args=<optimized out>, nargs=0, keywords=0x0) at Objects/call.c:214
#12 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#13 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df86c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#14 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#15 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x7f479d5b6640, tuple=0x7f47a6d8bb18 <_PyRuntime+58904>, callable=0x7f479ec9ade0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#16 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f479ec9ade0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=0x7f479d5b6640) at Objects/call.c:328
#17 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x7f479d5b6640, callargs=0x7f47a6d8bb18 <_PyRuntime+58904>, func=0x7f479ec9ade0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#19 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df8608, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#20 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#21 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479f865e70, callable=0x7f47a58b3560, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#22 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a58b3560, args=0x7f479f865e70, kwargs=0x0) at Objects/call.c:328
#23 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479f865e70, func=0x7f47a58b3560, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#24 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#25 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df83a0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#26 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#27 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641abd0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479d5b69c0) at Objects/call.c:152
#28 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b510, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479d5b69c0) at Objects/call.c:482
#29 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b510, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479d5b69c0) at Objects/typeobject.c:7630
#30 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b510, args=<optimized out>, nargs=0, keywords=0x7f47a58bdc00) at Objects/call.c:214
#31 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#32 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df82c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#33 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#34 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479e434040, callable=0x7f47a59dca40, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#35 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dca40, args=0x7f479e434040, kwargs=0x0) at Objects/call.c:328
#36 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479e434040, func=0x7f47a59dca40, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#38 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df8058, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#39 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#40 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641b0c0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479ecb46c0) at Objects/call.c:152
#41 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b6f0, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479ecb46c0) at Objects/call.c:482
#42 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b6f0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479ecb46c0) at Objects/typeobject.c:7630
#43 0x00007f47a6969728 in _PyObject_Call (kwargs=0x7f479ecb46c0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, callable=0x7f47a576b6f0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:343
#44 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b6f0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=0x7f479ecb46c0) at Objects/call.c:313
#45 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x7f479ecb46c0, callargs=0x7f47a6d8bb18 <_PyRuntime+58904>, func=0x7f47a576b6f0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#46 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#47 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7cf8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#48 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#49 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479e3b6900, callable=0x7f47a59dc7c0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#50 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dc7c0, args=0x7f479e3b6900, kwargs=0x0) at Objects/call.c:328
#51 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479e3b6900, func=0x7f47a59dc7c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#52 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#53 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7a90, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#54 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#55 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641b5c0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479eca3640) at Objects/call.c:152
#56 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b880, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479eca3640) at Objects/call.c:482
#57 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b880, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479eca3640) at Objects/typeobject.c:7630
#58 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b880, args=<optimized out>, nargs=0, keywords=0x7f47a5950300) at Objects/call.c:214
#59 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#60 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df79f8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
[4:15](https://bluebrainproject.slack.com/archives/C071MBESA6R/p1714659301772289)
#61 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#62 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a4cda170, callable=0x7f47a59dd8a0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#63 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dd8a0, args=0x7f47a4cda170, kwargs=0x0) at Objects/call.c:328
#64 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f47a4cda170, func=0x7f47a59dd8a0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#65 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#66 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7790, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#67 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#68 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641bab0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f47a4cbda00) at Objects/call.c:152
#69 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b970, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f47a4cbda00) at Objects/call.c:482
#70 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b970, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f47a4cbda00) at Objects/typeobject.c:7630
#71 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b970, args=<optimized out>, nargs=0, keywords=0x7f47a5f4f700) at Objects/call.c:214
#72 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#73 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df75e8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#74 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#75 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a4fa5ea0, callable=0x7f47a59dd760, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#76 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dd760, args=0x7f47a4fa5ea0, kwargs=0x0) at Objects/call.c:328
#77 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f47a4fa5ea0, func=0x7f47a59dd760, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#78 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#79 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7380, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#80 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#81 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641bfa0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f47a4c489c0) at Objects/call.c:152
#82 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576ac50, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f47a4c489c0) at Objects/call.c:482
#83 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576ac50, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f47a4c489c0) at Objects/typeobject.c:7630
#84 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576ac50, args=<optimized out>, nargs=0, keywords=0x7f47a5f4fd90) at Objects/call.c:214
#85 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#86 0x00007f47a6a75925 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df71b8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#87 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7f47a643b880, func=0x7f47a62a2520, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:6428
#88 PyEval_EvalCode (co=co@entry=0x7f47a56eb440, globals=globals@entry=0x7f47a643b880, locals=locals@entry=0x7f47a643b880) at Python/ceval.c:1154
#89 0x00007f47a6a71740 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7f47a643b880, globals=0x7f47a643b880, source=0x7f47a56eb440) at Python/bltinmodule.c:1075
#90 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#91 0x00007f47a69bea4e in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f47a63d0fe0, args=0x7f47a6df7180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:443
#92 0x00007f47a696a088 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7f47a63d0fe0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#93 PyObject_Vectorcall (callable=0x7f47a63d0fe0, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:299
#94 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#95 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7020, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#96 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#97 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a6241a00, callable=0x7f47a62a2200, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#98 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a62a2200, args=0x7f47a6241a00, kwargs=0x0) at Objects/call.c:328
#99 0x00007f47a6ae3239 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#100 0x00007f47a6ae3afa in pymain_run_python (exitcode=0x7fffc641c5b0) at Modules/main.c:595
#101 Py_RunMain () at Modules/main.c:680
#102 0x00007f47a6ae41ea in pymain_main (args=0x7fffc641c6d0) at Modules/main.c:710
#103 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#104 0x00007f47a6639083 in __libc_start_main (main=0x55d9d8329060 <main>, argc=5, argv=0x7fffc641c838, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffc641c828) at ../csu/libc-start.c:308
#105 0x000055d9d832909e in _start ()
@mgeplf mgeplf added the bug label May 2, 2024
@mgeplf
Copy link
Collaborator Author

mgeplf commented May 5, 2024

My current theory is that:

  1. Since there is a single invocation of the python interpreter, there are interactions between the tests being run
  2. There is global state being stored, specifically sgid2srcindex_ and visources_ in src/nrniv/partrans.cpp
  3. test/pytest_coreneuron/test_partrans.py cause the above globals to be modified - the Node's allocated and stored get free'd, but the references in the globals aren't cleared.
  4. Non-deterministically, a later test overwrites some heap memory
  5. Later, in mk_ttd, the dangling Node ref is used: https://github.com/neuronsimulator/nrn/blob/master/src/nrniv/partrans.cpp#L499)
  6. Depending on the overwrite in 4), there is a segfault

When I have some more time, I'll see if I can confirm the theory

@mgeplf
Copy link
Collaborator Author

mgeplf commented May 27, 2024

Got a valgrind trace that summarizes the above:

test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py ==21626== Invalid read of size 8
==21626==    at 0x8EDD150: mk_ttd() (partrans.cpp:503)
==21626==    by 0x8F49D26: nrn_thread_memblist_setup() (multicore.cpp:647)
==21626==    by 0x8F406DC: v_setup_vectors() (treeset.cpp:1697)
==21626==    by 0x8F37912: nrnhoc_topology() (solve.cpp:296)
==21626==    by 0x8F530E6: hoc_call() (code.cpp:1418)
==21626==    by 0x8FDFF19: fcall(void*, void*) (nrnpy_hoc.cpp:728)
==21626==    by 0x8ED9ACD: OcJump::fpycall(void* (*)(void*, void*), void*, void*) (ocjump.cpp:138)
==21626==    by 0x8FDCF09: hocobj_call(PyHocObject*, _object*, _object*) (nrnpy_hoc.cpp:796)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49AC788: _PyVectorcall_Call (call.c:245)
==21626==    by 0x49AC788: _PyObject_Call (call.c:328)
==21626==  Address 0x8676a98 is 104 bytes inside a block of size 120 free'd
==21626==    at 0x483D1CF: operator delete(void*, unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==21626==    by 0x8F37B85: node_destruct(Node**, int) (solve.cpp:585)
==21626==    by 0x8F39597: node_free (solve.cpp:474)
==21626==    by 0x8F39597: sec_free(hoc_Item*) (solve.cpp:502)
==21626==    by 0x8F05D47: delete_section() (cabcode.cpp:358)
==21626==    by 0x8F530E6: hoc_call() (code.cpp:1418)
==21626==    by 0x8FDFF19: fcall(void*, void*) (nrnpy_hoc.cpp:728)
==21626==    by 0x8ED9ACD: OcJump::fpycall(void* (*)(void*, void*), void*, void*) (ocjump.cpp:138)
==21626==    by 0x8FDCF09: hocobj_call(PyHocObject*, _object*, _object*) (nrnpy_hoc.cpp:796)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49AC788: _PyVectorcall_Call (call.c:245)
==21626==    by 0x49AC788: _PyObject_Call (call.c:328)
==21626==  Block was alloc'd at
==21626==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==21626==    by 0x8F37D48: node_alloc(Section*, short) (solve.cpp:744)
==21626==    by 0x8F02E30: nrn_change_nseg(Section*, int) (cabcode.cpp:1508)
==21626==    by 0x8F03977: new_section(Object*, Symbol*, int) (cabcode.cpp:298)
==21626==    by 0x8F0467F: nrnpy_newsection(NPySecObj*) (cabcode.cpp:327)
==21626==    by 0x8FE7A99: NPySecObj_init(NPySecObj*, _object*, _object*) (nrnpy_nrn.cpp:376)
==21626==    by 0x8FE7B8C: NPySecObj_new(_typeobject*, _object*, _object*) (nrnpy_nrn.cpp:401)
==21626==    by 0x4A013E2: cfunction_call (methodobject.c:542)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49ACB7A: _PyObject_FastCallDictTstate (call.c:141)

The reduced set of tests required to trigger it somewhat often (1 in 10 times?) is:

python3 -m pytest \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_basic.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_bbss.py \
    \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_coreneuron_configuration.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_hoc_po.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_nrntest_fast.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_partrans.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py \

@pramodk
Copy link
Member

pramodk commented May 28, 2024

@mgeplf : I haven't looked at the code details but not surprised by the global state-related issue. With the code that you have skimmed through, do you know already a potential fix?

@mgeplf
Copy link
Collaborator Author

mgeplf commented May 29, 2024

My guess is that one of node_destruct/node_free/sec_free/delete_section has to be made aware of partrans.cpp::visources_.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pramodk @mgeplf