Make module loader in multithreading mode be more aware when init fails

[tanishiking]

#3591

• Catch an exception thrown in ctor(instance) (in module_load.cpp)
• Store the thrown exception somehow and notify to other threads that module initialization is failed
  • Through InitializationContext
  • Waiting (for module initialization) threads will know when the initialization is failed in a thread via InitializationContext.
• Show meaningful error/stacktrace to users

[tanishiking]

Show meaningful error/stacktrace to users

• Currently, we just re-throw the C++ exception, and die with libc++abi: terminating due to uncaught exception of type void*.
• Maybe we need to store the thrown exception into pointer, and re-throw in Scala world? Potentially related "We potentially can provide a CFuncPtr to the native code which would be used to wrap the exception thrown by the ctor into the ExceptionInInitializerError, which later can be thrown using the scalanative_throw" Loading module in multithreading mode should be more aware when init fails #3591 (comment)
• It should be around here and can pass something like val exceptionSlot = Val.Local(fresh(), Type.Ptr) to __scalanative_loadModule but should we re-throw in NIR level? -> oh, wait we can just call Scala function from NIR...
• update I talked with @WojciechMazur and we would optimize error handling during initialization by introducing a custom method within codegen.Generate that takes an object, creates an initialization error instance, and throws it.
• 9e1f8e2 generates a function that calls another Scala function that throws ExceptionInInitializerError in codegen.Generate. Now module_load.cpp calls that generated function.
  • However, we now get the following runtime error.
  • Since we didn't see throw exception in initializer error, it seems we didn't even reach the throwExceptionInInitializerError function defined in Scala 🤔

❯ lldb ./sandbox/.3/target/scala-3.3.1/sandbox
(lldb) target create "./sandbox/.3/target/scala-3.3.1/sandbox"
Current executable set to '/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/sandbox' (arm64).
(lldb) run
Process 93413 launched: '/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/sandbox' (arm64)
start thread 0start thread 0start thread 3
start init by Thread[5,thread-3,5,main]

tart init by Thread[5,thread-3,5,main]d 5
start thread 2
start thread 4
start thread 6

t

tart thread 6start thread 1
start thread 7
Process 93413 stopped
* thread #7, stop reason = signal SIGBUS
    frame #0: 0x00000001001c44ac sandbox`scalanative_gc_safepoint_poll at ImmixGC.c:149:20
   146      MutatorThread_switchState(currentMutatorThread, state);
   147  }
   148  void scalanative_gc_safepoint_poll() {
-> 149      void *pollGC = *scalanative_gc_safepoint;
   150  }
   151
   152  void scalanative_add_roots(void *addr_low, void *addr_high) {

(lldb) bt
* thread #7, stop reason = signal SIGBUS
  * frame #0: 0x00000001001c44ac sandbox`scalanative_gc_safepoint_poll at ImmixGC.c:149:20
    frame #1: 0x00000001001c6cd4 sandbox`waitForInitialization(slot=0x0000000100323940, classInfo=0x0000000100325d00) at module_load.cpp:94:9
    frame #2: 0x00000001001c6bc4 sandbox`::__scalanative_loadModule(slot=0x0000000100323940, classInfo=0x0000000100325d00, size=24, ctor=(sandbox`_SM12Deadlocking$RE at Test.scala)) at module_load.cpp:134:16
    frame #3: 0x0000000100050d20 sandbox`_SM12Deadlocking$G4load at Test.scala:1:8
    frame #4: 0x0000000100050fd8 sandbox`_SM5Test$D10$anonfun$1iuEPT5Test$(n$1=5) at Test.scala:16:41 [opt]
    frame #5: 0x00000001000518cc sandbox`_SM15Test$$$Lambda$4D3runuEO at Test.scala:18:10 [opt]
    frame #6: 0x0000000100140780 sandbox`_SM16java.lang.ThreadD3runuEO(this=0x00000002c000e280) at Thread.scala:247:31 [opt]
    frame #7: 0x000000010002bad8 sandbox`_SM39scala.scalanative.runtime.NativeThread$D16threadEntryPointL38scala.scalanative.runtime.NativeThreaduEPT39scala.scalanative.runtime.NativeThread$(this=0x0000000100321248, nativeThread=0x00000002c0040c90) at NativeThread.scala:133:19
    frame #8: 0x000000010002c578 sandbox`_SM39scala.scalanative.runtime.NativeThread$D24threadRoutine$$anonfun$1L28scala.scalanative.unsafe.PtrL28scala.scalanative.unsafe.PtrEPT39scala.scalanative.runtime.NativeThread$(arg=<unavailable>) at NativeThread.scala:121:36 [opt]
    frame #9: 0x000000010002c5ec sandbox`_SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder at NativeThread.scala:117:71 [opt]
    frame #10: 0x00000001001c4448 sandbox`ProxyThreadStartRoutine(args=0x0000600001ac0010) at ImmixGC.c:113:5
    frame #11: 0x0000000182ba7034


(lldb) fr va
(void *) pollGC = 0x00000001001c6ccc

[tanishiking]

Talked with @WojciechMazur. The SIGBUS (or SIGSEGV) signals are deliberately triggered to create safepoints for GC (by attempting to access 0x0 that's what scalanative_gc_safepoint_poll() does).
Therefore, debugger naturally gonna pause on the scalanative_gc_safepoint_poll(), and it's not the real problem here. In lldb we can ignore SIGBUS by pro hand -p true -s false SIGBUS.

We tried to re-throw an exception (in Scala, calling from module_load.cpp) but we got the following error and died with segmentation fault.

Unexpected signal 10 when accessing memory at address 0x104c08588
        at StackTrace_PrintStackTrace
        at SafepointTrapHandler
        at _sigtramp
zsh: segmentation fault  

(My two cent: this is probably caused because the memory address for module in waiting threads are freed by the main thread (that initialize the module ... and fails) when it failed, and those waiting threads end up accessing freed memory addresses?)

---

Anyway, 24d5ad0 has kind of improvement

• Now, both waitForInitialization and __scalanative_loadModule returns nullptr when initialization failed. The thread initialize the module will notify to other threads by
  • Catch an exception thrown by ctor(instance)
  • Then assign nullptr to module and return nullptr.
  • Other threads will read the module and when it is nullptr, they interpret that the module initialization is failed. (it should points to InitializationContext while waiting for module initialization).
  • Then those threads also will return nullptr.
• If the return value of load was nullptr, the generated code in genModuleOp will throw an exception.
• This exception will be caught by

  scala-native/nativelib/src/main/scala/scala/scalanative/runtime/NativeThread.scala

  Lines 133 to 141 in 644fbd5

  	try thread.run()
  	catch {
  	case ex: Throwable =>
  	val handler = thread.getUncaughtExceptionHandler() match {
  	case null => Thread.getDefaultUncaughtExceptionHandler()
  	case handler => handler
  	}
  	if (handler != null)
  	executeUncaughtExceptionHandler(handler, thread, ex)

While this will add up a bit more instructions to the generated code (return value check for every module initialization), it anyway works.

We need some more improvements

• Throw NoClassDefFoundError or ExceptionInInitializerError instead of NullPointerException (which is thrown for a time being).
  • In JVM, the main thread throws ExceptionInInitializerError while others throw NoClassDefFoundError.
  • Those exception should have an original exception thrown while module initialization in cause field

Result of `sandbox3 / run`, stacktrace is a bit messy 😢

start thread 0

start thread 3

start thread 3tart thread 2
start thread 4
start init by Thread[2,thread-0,5,main]
start thread 5
start thread 6
start thread 7
start thread 1
Exception in module initializer: std::exception
Exception in module initializer wait
Exception in module initializer wait
exception! java.lang.NoSuchMethodException: test
Exception in thread "thread-0"java.lang.NoSuchMethodException: test
        at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)
        at Test$.$anonfun$1(Test.scala:15)
exception! java.lang.NoSuchMethodException: test
        at Test$$$Lambda$4.run(Test.scala:19)

Exception in thread "thread-7"  at .laag.NoS.chMeadodEx(Tptiod. test:245)
Exception in module initializer wait
        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)
        at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)
        at <none>._SM49scala.scalanative.runtime.       at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)
        at <none>._pthread_start(Unknown Source)
        at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)
        at Test$.$anonfun$1(Test.scala:15)
        at Test$$$Lambda$4.run(Test.scala:19)
        at java.lang.Thread.run(Thread.scala:245)
        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)
exception! java.lang.NoSuchMethodException: test        at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)Exception in module initializer wait

        at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)exception! java.lang.NoSuchMethodException: testexception! java.lang.NoSuchMethodException: testException in module initializer wait



Exception in thread "thread-1"
Exception in thread "thread-1"Exception in thread "thread-5"java.lang.NoSuchMethodException: testjava.lang.NoSuchMethodException: testjava.lang.NoSuchMethodException: test


        at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)   at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)


        at Test$.$anonfun$1(Test.scala:15)      at Test$.$anonfun$1(Test.scala:15)


        at Test$$$Lambda$4.run(Test.scala:19)
exception! java.lang.NoSuchMethodException: test

        Exception in thread "thread-2"d.scala:245)Exception in thread "thread-2"Exception in thread "thread-2"       at <none>._pthread_start(Unknown Source)
xception in thread "thread-2"   at <none>._pthread_start(Unknown Source)


java.lang.NoSuchMethodException: test
        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)Exception in module initializer wait


        at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)


        at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118) at Test$.$anonfun$1(Test.scala:15)exception! java.lang.NoSuchMethodException: test


exception! java.lang.NoSuchMethodException: test
Exception in thread "thread-4"Exception in thread "thread-4"Exception in thread "thread-3"java.lang.NoSuchMethodException: test

ava.lang.NoSuchMethodException: test
ava.lang.NoSuchMethodException: test
ava.lang.NoSuchMethodException: test)
at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)

Exception in module initializer wait
        at Test$.$anonfun$1(Test.scala:15)ge    at Test$.$anonfun$1(Test.scala:15)


        at <none>._pthread_start(Unknown Sour
at <none>._pthread_start(Unknown Source)


        at java.lang.Thread.run(Thread.scala:245)



        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)
        at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)
        at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/sca
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/sca

        at java.lang.Thread.run(Thread.scala:245)

at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)

at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)exception! java.lang.NoSuchMethodException: test


        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)
        at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)Exception in thread "thread-6"
java.lang.NoSuchMethodException: test


        at Test$.$anonfun$1(Test.scala:15)sthread$.threadEntryPoint(NativeThread.scala:126)Exception in thread "thread-6"    at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)c:105)


        at scala.scalanative.runtime.package$.throwNoSuchMethod(package.scala:159)

        at Test$.$anonfun$1(Test.scala:15)
        at <nst$$$Lambda$4.run(Test.scala:19)   at <nst$$$Lambda$4.run(Test.scala:19)        at Test$$$Lambda$4.run(Test.scala:19)   at <nst$$$Lambda$4.run(Test.scala:19)        at Test$$$Lambda$4.run(Test.scala:19)king/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)


at java.lang.Thread.run(Thread.scala:245)
a

        at <none>._pthread_start(Unknown Source)        at <none>._pthread_start(Unknown Source)     at scala.scalanative.runtime.NativeThread$.threadEntryPoint(NativeThread.scala:126)

                at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
                at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
        at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)
                at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)


        at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)
a
at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)
at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)

        at <none>._pthread_start(Unknown Source)


        at scala.scalanative.runtime.NativeThrea        at scala.scalanative.runtime.NativeThread$.threadRoutine$$anonfun$1(NativeThread.scala:119)

        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)

        at <none>._SM49scala.scalanative.runtime.NativeThread$$$Lambda$1G17$extern$forwarder(NativeThread.scala:118)

        at <none>._pthread_start(Unknown Source)
        at <none>.ProxyThreadStartRoutine(/Users/tanishiking/ghq/github.com/tanishiking/scala-native/sandbox/.3/target/scala-3.3.1/native/dependencies/nativelib_native0.5.0-SNAPSHOT_3-0.5.0-SNAPSHOT-0/scala-native/gc/immix/ImmixGC.c:105)
        at <none>._pthread_start(Unknown Source)
[success] Total time: 9 s, completed Nov 8, 2023, 1:57:10 PM