Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some Java DaCapo benchmarks fail with unhandled SIGSEGVs in hw mode #645

Closed
prp opened this issue Jul 19, 2020 · 3 comments · May be fixed by #646
Closed

Some Java DaCapo benchmarks fail with unhandled SIGSEGVs in hw mode #645

prp opened this issue Jul 19, 2020 · 3 comments · May be fixed by #646
Labels
area: sgx-lkl Core SGX-LKL functionality bug p1 Medium priority
Milestone
Milestone 1

Comments

@prp
Copy link
Member

prp commented Jul 19, 2020 •
edited

When running the Java DaCapo benchmarks, some of them fail with unhandled SIGSEGVs as follows:

$ SGXLKL_VERBOSE=1 SGXLKL_KERNEL_VERBOSE=1 SGXLKL_MMAP_FILES=Shared SGXLKL_TRACE_SIGNAL=1 sgx-lkl-run-oe --hw-debug --enclave-config=enclave_config-java-dacapo.json java-dacapo.img 
[   SGX-LKL  ] nproc=8 ETHREADS=8 CMDLINE="mem=32M"
[   SGX-LKL  ] HW TLS support: econf->fsgsbase=1
[   SGX-LKL  ] Registering disk 0 (path='java-dacapo.img', mnt='/', [RW])
[   SGX-LKL  ] No tap device specified, networking will not be available.
[   SGX-LKL  ] get_signed_libsgxlkl_path... result=/opt/sgx-lkl/bin/../lib/libsgxlkl.so.signed
[   SGX-LKL  ] oe_create_enclave...
[   SGX-LKL  ] Enclave config: {"args":["/usr/bin/java","-Xms2000k","-XX:InitialCodeCacheSize=2000k","-XX:ReservedCodeCacheSize=4000K","-XX:CompressedClassSpaceSize=4000K","-XX:+UseCompressedClassPointers","-XX:+PerfDisableSharedMem","-XX:+UseMembar","-Dsun.zip.disableMemoryMapping=true","-verbose:gc","-jar","/dacapo/dacapo-9.12-MR1-bach.jar","pmd"],"clock_res":[{"resolution":"0000000000000001"},{"resolution":"0000000000000001"},{"resolution":"0000000000000000"},{"resolution":"0000000000000000"},{"resolution":"0000000000000001"},{"resolution":"00000000003d0900"},{"resolution":"00000000003d0900"},{"resolution":"0000000000000001"}],"cwd":"/","env":["LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/lib/jvm/java-1.8-openjdk/lib/amd64/jli"],"esleep":"16000","espins":"500","ethreads":"8","exit_status":"full","format_version":"1","fsgsbase":true,"host_import_env":[],"hostname":"lkl","hostnet":false,"image_sizes":{"num_heap_pages":"2097152","num_stack_pages":"1024"},"io":{"block":true,"console":true,"network":true},"kernel_cmd":"mem=32M","kernel_verbose":true,"max_user_threads":"256","mmap_files":"shared","mode":"hw_debug","mounts":[],"net_gw4":"10.0.1.254","net_ip4":"10.0.1.1","net_mask4":"24","oe_heap_pagecount":"8192","root":{"key":"","key_id":null,"overlay":false,"readonly":false,"roothash":null,"roothash_offset":"0"},"stacksize":"524288","swiotlb":true,"sysctl":null,"tap_mtu":"0","verbose":true,"wg":{"ip":"10.0.2.1","key":null,"listen_port":"56002","peers":[]}}
result=0 (OE_OK)
[   SGX-LKL  ] sgxlkl_enclave_init(ethread_id=0)
[[  SGX-LKL ]] sgxlkl_enclave_init(): enter
[[  SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave base=0x7f3000000000 size=64.0 GiB
[[  SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave heap base=0x7f3000b47000 size=8.0 GiB end=0x7f3200b47000
[[  SGX-LKL ]] sgxlkl_enclave_init(): calling _dlstart_c()
[[  SGX-LKL ]] _register_enclave_signal_handlers(): Registering OE exception handler...
[[  SGX-LKL ]] __libc_init_enclave(): calling enclave_mman_init()
[[  SGX-LKL ]] __libc_init_enclave(): calling _lthread_sched_init()
[[  SGX-LKL ]] lkl_start_init(): calling register_lkl_syscall_overrides()
[[  SGX-LKL ]] lkl_start_init(): calling initialize_enclave_event_channel()
[[  SGX-LKL ]] lkl_start_init(): kernel command line: 'mem=32M console=hvc0'
[[  SGX-LKL ]] lkl_start_init(): Disk 0: Disk encryption: no
[[  SGX-LKL ]] lkl_start_init(): Disk 0: Disk is writable: yes
[[  SGX-LKL ]] lkl_start_init(): lkl_start_kernel() called
[    0.000000] Linux version 5.3.0+ (prp@dc-peter-dev) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #4 Sun Jul 19 14:34:37 UTC 2020
[    0.000000] memblock address range: 0x7f31fe8c6000 - 0x7f32008c6000
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8080
[    0.000000] Kernel command line: mem=32M console=hvc0 virtio_mmio.device=256@0x1000000:1
[    0.000000] Dentry cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory available: 32176k/32768k RAM
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 4096
[    0.000000] lkl: irqs initialized
[    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] lkl: time and timers initialized (irq2)
[    0.000557] printk: console [hvc0] enabled
[    0.000557] pid_max: default: 4096 minimum: 301
[    0.000557] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.000558] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.002761] devtmpfs: initialized
[    0.005442] random: get_random_bytes called from 0x00007f300066284b with crng_init=0
[    0.005444] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.005444] futex hash table entries: 16 (order: -4, 384 bytes, linear)
[    0.005445] xor: automatically using best checksumming function   8regs
[    0.005999] NET: Registered protocol family 16
[    0.011042] clocksource: Switched to clocksource lkl
[    0.011575] NET: Registered protocol family 2
[    0.012135] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.012136] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.012137] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.012685] TCP: Hash tables configured (established 512 bind 512)
[    0.012686] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.012686] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.012687] NET: Registered protocol family 1
[    0.012687] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x10000ff, IRQ 1.
[    0.013241] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    0.014333] NET: Registered protocol family 38
[    0.014333] io scheduler mq-deadline registered
[    0.014888] io scheduler kyber registered
[    0.094890] software IO TLB: mapped [mem 0x7f415d7ba000-0x7f41617ba000] (64MB)
[    0.096917] loop: module loaded
[    0.097422] device-mapper: ioctl: 4.40.0-ioctl (2019-01-18) initialised: dm-devel@redhat.com
[    0.097927] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[    0.098482] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[    0.098484] NET: Registered protocol family 10
[    0.099594] Segment Routing with IPv6
[    0.099594] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.101340] This architecture does not have kernel memory protection.
[    0.102511] Run /init as init process
[[  SGX-LKL ]] lkl_start_init(): lkl_start_kernel() finished
[[  SGX-LKL ]] lkl_start_init(): creating LKL termination thread
[[  SGX-LKL ]] create_lkl_termination_thread(): enter
[[  SGX-LKL ]] lkl_start_init(): calling lkl_mount_virtual()
[[  SGX-LKL ]] lkl_termination_thread(): enter
[[  SGX-LKL ]] lkl_start_init(): calling init_random()
[[  SGX-LKL ]] init_random(): Adding entropy to entropy pool
[[  SGX-LKL ]] lkl_termination_thread(): Performed LKL syscall to get host task allocated (pid=36)
[    0.104191] random: crng init done
[[  SGX-LKL ]] init_enclave_clock(): Setting enclave realtime clock
[[  SGX-LKL ]] wg0 has public key +LaycJKfiHfGfSX5HagcZpomYOTX1pcX6Ilzv9DPdw8=
[    0.108634] virtio_blk virtio1: [vda] 1024000 512-byte logical blocks (524 MB/500 MiB)
[[  SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vda", mnt="/mnt/vda", ro=0)
[    0.110864] EXT4-fs (vda): mounted filesystem without journal. Opts:
[[  SGX-LKL ]] lkl_mount_disks(): Set working directory: /
[[  SGX-LKL ]] libc_start_main_stage2(): Calling app main: /usr/bin/java
OpenJDK 64-Bit Server VM warning: Can't detect primordial thread stack location - find_vma failed
[GC (Allocation Failure)  512K->496K(1536K), 0.0044627 secs]
[GC (Allocation Failure)  1008K->550K(1536K), 0.0037532 secs]
[GC (Allocation Failure)  1059K->618K(1536K), 0.0011175 secs]
[GC (Allocation Failure)  1130K->813K(2048K), 0.0016743 secs]
[GC (Allocation Failure)  1837K->927K(2048K), 0.0137295 secs]
[Full GC (Ergonomics)  927K->571K(3072K), 0.0116470 secs]
[GC (Allocation Failure)  1595K->854K(4608K), 0.0022363 secs]
[GC (Allocation Failure)  2902K->900K(4608K), 0.0108369 secs]
[GC (Allocation Failure)  2943K->902K(6656K), 0.0015773 secs]
[GC (Allocation Failure)  4998K->982K(6656K), 0.0100926 secs]
[GC (Allocation Failure)  5063K->930K(9216K), 0.0011219 secs]
[GC (Allocation Failure)  7584K->922K(9216K), 0.0150401 secs]
[GC (Allocation Failure)  7528K->904K(13312K), 0.0011162 secs]
[GC (System.gc())  1169K->910K(13312K), 0.0011134 secs]
[Full GC (System.gc())  910K->599K(13312K), 0.0061311 secs]
===== DaCapo 9.12-MR1 pmd starting =====
[GC (Allocation Failure)  11351K->3741K(13824K), 0.0710138 secs]
[Full GC (Ergonomics)  3741K->3431K(16384K), 0.0585798 secs]
[GC (Allocation Failure)  13159K->7022K(17408K), 0.0055747 secs]
[Full GC (Ergonomics)  7022K->3926K(19968K), 0.0216754 secs]
[GC (Allocation Failure)  13654K->5121K(24064K), 0.0040992 secs]
[GC (Allocation Failure)  17921K->6081K(24064K), 0.0153990 secs]
[GC (Allocation Failure)  18881K->7086K(27136K), 0.0100002 secs]
[GC (Allocation Failure)  22446K->7575K(27648K), 0.0313323 secs]
[GC (Allocation Failure)  22935K->8251K(31744K), 0.0061429 secs]
[GC (Allocation Failure)  27707K->14219K(36864K), 0.0787700 secs]
[GC (Allocation Failure)  33675K->21925K(48128K), 0.0874111 secs]
[Full GC (Ergonomics)  21925K->15860K(52736K), 0.1773813 secs]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f31fcf5087e, pid=36, tid=0x00007f3000b50a00
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.15.0
# Distribution: Custom build (Wed Jan 29 10:50:51 UTC 2020)
# Problematic frame:
# J 1094 C1 net.sourceforge.pmd.ast.JavaParser.jj_3R_117()Z (88 bytes) @ 0x00007f31fcf5087e [0x00007f31fcf502c0+0x5be]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
bad count while changing owner (cpu.count=2 cpu.owner=139844147022336 owner=139844147019296)
[[  SGX-LKL ]] FAIL: Kernel panic! Aborting...
[[  SGX-LKL ]] =============================================================
[[  SGX-LKL ]] Stack traces for all lthreads:
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 1: tid=1 (7f3000b489d0) [java]
[[  SGX-LKL ]]     #0: 7f30005b36be in lthread_join(...)
[[  SGX-LKL ]]     #1: 7f3000599168 in __pthread_join(...)
[[  SGX-LKL ]]     #2: 7f31fdd36ece in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 2: tid=2 (7f3000b49de0) [vio-0]
[[  SGX-LKL ]]     #0: 7f30005a5696 in vio_wait_for_host_event(...)
[[  SGX-LKL ]]     #1: 7f30005a5866 in vio_enclave_process_host_event(...)
[[  SGX-LKL ]]     #2: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 3: tid=3 (7f3000b49fa0) [vio-1]
[[  SGX-LKL ]]     #0: 7f30005a5696 in vio_wait_for_host_event(...)
[[  SGX-LKL ]]     #1: 7f30005a5866 in vio_enclave_process_host_event(...)
[[  SGX-LKL ]]     #2: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 4: tid=4 (7f3000b4a160) [vio-2]
[[  SGX-LKL ]]     #0: 7f30005a5696 in vio_wait_for_host_event(...)
[[  SGX-LKL ]]     #1: 7f30005a5866 in vio_enclave_process_host_event(...)
[[  SGX-LKL ]]     #2: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 5: tid=5 (7f3000b4a320) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f300041254d in schedule_idle(...)
[[  SGX-LKL ]]     #8: 7f30000ae37a in do_idle(...)
[[  SGX-LKL ]]     #9: 7f30000ae5d5 in cpu_startup_entry(...)
[[  SGX-LKL ]]     #10: 7f30004102be in rest_init(...)
[[  SGX-LKL ]]     #11: 7f3000657988 in arch_call_rest_init(...)
[[  SGX-LKL ]]     #12: 7f3000657d2b in start_kernel(...)
[[  SGX-LKL ]]     #13: 7f3000658616 in lkl_run_kernel(...)
[[  SGX-LKL ]]     #14: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 6: tid=6 (7f3000b4a4e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b469a in mutex_lock(...)
[[  SGX-LKL ]]     #5: 7f30000892da in __cpu_try_get_lock(...)
[[  SGX-LKL ]]     #6: 7f30000896c4 in lkl_cpu_try_run_irq(...)
[[  SGX-LKL ]]     #7: 7f30000887dc in lkl_trigger_irq(...)
[[  SGX-LKL ]]     #8: 7f300008896f in timer_fn(...)
[[  SGX-LKL ]]     #9: 7f30005b4c34 in timer_callback(...)
[[  SGX-LKL ]]     #10: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 7: tid=7 (7f3000b4a6a0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005b3516 in lthread_join(...)
[[  SGX-LKL ]]     #1: 7f30005b4950 in thread_exit(...)
[[  SGX-LKL ]]     #2: 7f3000087c5e in lkl_run_init(...)
[[  SGX-LKL ]]     #3: 7f300010d55f in search_binary_handler(...)
[[  SGX-LKL ]]     #4: 7f300010db65 in __do_execve_file(...)
[[  SGX-LKL ]]     #5: 7f300010dc9d in do_execve(...)
[[  SGX-LKL ]]     #6: 7f3000087602 in run_init_process(...)
[[  SGX-LKL ]]     #7: 7f30004102fc in kernel_init(...)
[[  SGX-LKL ]]     #8: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #9: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 8: tid=8 (7f3000b4a860) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a6b45 in kthreadd(...)
[[  SGX-LKL ]]     #9: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #10: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 9: tid=9 (7f3000b4aa20) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 10: tid=10 (7f3000b4abe0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 11: tid=11 (7f3000b4ade0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 12: tid=12 (7f3000b4afe0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 13: tid=13 (7f3000b4b1e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a9c95 in smpboot_thread_fn(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 14: tid=14 (7f3000b4b3e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f3000252726 in devtmpfsd(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 15: tid=15 (7f3000b4b5e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 16: tid=34 (7f3000b4b7e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 17: tid=35 (7f3000b4b9e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 18: tid=36 (7f3000b4bbe0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000ebb54 in kswapd(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 19: tid=37 (7f3000b4bde0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 20: tid=38 (7f3000b4bfe0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f300023c815 in khvcd(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 21: tid=39 (7f3000b4c1e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 22: tid=40 (7f3000b4c3e0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 23: tid=41 (7f3000b4c630) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f3000088bdf in idle_host_task_loop(...)
[[  SGX-LKL ]]     #6: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #7: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 24: tid=42 (7f3000b4c830) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f30005b822b in lkl_termination_thread(...)
[[  SGX-LKL ]]     #6: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 25: tid=43 (7f3000b4cce0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 26: tid=44 (7f3000b4caf0) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 27: tid=45 (7f3000b4cf50) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1303 in worker_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 28: tid=46 (7f3000b4d150) [kernel]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000a1eb6 in rescuer_thread(...)
[[  SGX-LKL ]]     #9: 7f30000a5fd4 in kthread(...)
[[  SGX-LKL ]]     #10: 7f3000087ef8 in thread_bootstrap(...)
[[  SGX-LKL ]]     #11: 7f30005b1fab in _exec(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 29: tid=47 (7f3000b4d330) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95fb94 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 30: tid=48 (7f3000b4d5c0) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 31: tid=49 (7f3000b4d820) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 32: tid=50 (7f3000b4da80) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 33: tid=51 (7f3000b4dce0) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 34: tid=52 (7f3000b4df40) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 35: tid=53 (7f3000b4e1a0) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 36: tid=54 (7f3000b4e400) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 37: tid=55 (7f3000b4e660) [java]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 38: tid=56 (7f3000b4e8c0) [VM Thread]
[[  SGX-LKL ]]     #0: 7f300057c7ed in sched_yield(...)
[[  SGX-LKL ]]     #1: 7f31fd9a3bf9 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 39: tid=57 (7f3000b4eb20) [Reference Handl]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 40: tid=58 (7f3000b4ed80) [Finalizer]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 41: tid=59 (7f3000b4efe0) [Signal Dispatch]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f300059baef in sem_timedwait(...)
[[  SGX-LKL ]]     #16: 7f300059bc0c in sem_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95a8e2 in <unknown>(...)
[[  SGX-LKL ]]     #18: 7f31765d2000 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 42: tid=60 (7f3000b4f240) [C2 CompilerThre]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 43: tid=61 (7f3000b4f4a0) [C2 CompilerThre]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b469a in mutex_lock(...)
[[  SGX-LKL ]]     #5: 7f30000892da in __cpu_try_get_lock(...)
[[  SGX-LKL ]]     #6: 7f3000089462 in lkl_cpu_get(...)
[[  SGX-LKL ]]     #7: 7f3000088da3 in lkl_syscall(...)
[[  SGX-LKL ]]     #8: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #9: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #10: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #11: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #12: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 44: tid=62 (7f3000b4f700) [C2 CompilerThre]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b469a in mutex_lock(...)
[[  SGX-LKL ]]     #5: 7f30000892da in __cpu_try_get_lock(...)
[[  SGX-LKL ]]     #6: 7f3000089462 in lkl_cpu_get(...)
[[  SGX-LKL ]]     #7: 7f3000088da3 in lkl_syscall(...)
[[  SGX-LKL ]]     #8: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #9: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #10: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #11: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #12: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 45: tid=63 (7f3000b4f960) [C1 CompilerThre]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f31fd95f769 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 46: tid=64 (7f3000b4fbc0) [Service Thread]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 47: tid=65 (7f3000b4fe20) [VM Periodic Tas]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f31fd95f769 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 48: tid=66 (7f3000b50080) [PmdThread 1]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 49: tid=67 (7f3000b502e0) [PmdThread 2]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 50: tid=68 (7f3000b50540) [PmdThread 3]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 51: tid=69 (7f3000b507a0) [PmdThread 4]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] *52: tid=70 (7f3000b50a00) [PmdThread 5]
[[  SGX-LKL ]]     #0: 7f30005a8e0e in sgxlkl_fail(...)
[[  SGX-LKL ]]     #1: 7f30005b4271 in panic(...)
[[  SGX-LKL ]]     #2: 7f30000827d3 in lkl_bug(...)
[[  SGX-LKL ]]     #3: 7f300008802c in __switch_to(...)
[[  SGX-LKL ]]     #4: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #5: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #6: 7f3000412805 in io_schedule(...)
[[  SGX-LKL ]]     #7: 7f3000412bf0 in bit_wait_io(...)
[[  SGX-LKL ]]     #8: 7f3000412889 in __wait_on_bit(...)
[[  SGX-LKL ]]     #9: 7f300041296e in out_of_line_wait_on_bit(...)
[[  SGX-LKL ]]     #10: 7f30001389fa in __wait_on_buffer(...)
[[  SGX-LKL ]]     #11: 7f300017e879 in ext4_read_inode_bitmap(...)
[[  SGX-LKL ]]     #12: 7f300017f197 in __ext4_new_inode(...)
[[  SGX-LKL ]]     #13: 7f30001a6182 in ext4_create(...)
[[  SGX-LKL ]]     #14: 7f3000114d0d in path_openat(...)
[[  SGX-LKL ]]     #15: 7f3000115313 in do_filp_open(...)
[[  SGX-LKL ]]     #16: 7f3000104a7f in do_sys_open(...)
[[  SGX-LKL ]]     #17: 7f3000104aff in __se_sys_openat(...)
[[  SGX-LKL ]]     #18: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #19: 7f300052e42e in __filter_syscall4(...)
[[  SGX-LKL ]]     #20: 7f300052e55e in open(...)
[[  SGX-LKL ]]     #21: 7f31fda1d2c2 in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 53: tid=71 (7f3000b50c60) [PmdThread 6]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 54: tid=72 (7f3000b50ec0) [PmdThread 7]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] -------------------------------------------------------------
[[  SGX-LKL ]] 55: tid=73 (7f3000b51120) [PmdThread 8]
[[  SGX-LKL ]]     #0: 7f30005aaf5c in __do_futex_sleep(...)
[[  SGX-LKL ]]     #1: 7f30005aaffc in futex_wait(...)
[[  SGX-LKL ]]     #2: 7f30005ab258 in enclave_futex(...)
[[  SGX-LKL ]]     #3: 7f30005b4419 in futex_wait(...)
[[  SGX-LKL ]]     #4: 7f30005b451a in sem_down(...)
[[  SGX-LKL ]]     #5: 7f300008805f in __switch_to(...)
[[  SGX-LKL ]]     #6: 7f300041228d in __schedule(...)
[[  SGX-LKL ]]     #7: 7f30004124b2 in schedule(...)
[[  SGX-LKL ]]     #8: 7f30000d3a01 in futex_wait_queue_me(...)
[[  SGX-LKL ]]     #9: 7f30000d53a9 in futex_wait(...)
[[  SGX-LKL ]]     #10: 7f30000d671b in do_futex(...)
[[  SGX-LKL ]]     #11: 7f30000d6ee5 in sys_futex(...)
[[  SGX-LKL ]]     #12: 7f3000088e1f in lkl_syscall(...)
[[  SGX-LKL ]]     #13: 7f30005964a5 in __filter_syscall6(...)
[[  SGX-LKL ]]     #14: 7f30005965bb in __timedwait_cp(...)
[[  SGX-LKL ]]     #15: 7f3000598a7e in __pthread_cond_timedwait(...)
[[  SGX-LKL ]]     #16: 7f3000598e20 in pthread_cond_wait(...)
[[  SGX-LKL ]]     #17: 7f31fd95f57b in <unknown>(...)
[[  SGX-LKL ]] =============================================================
2020-07-19T15:08:54.000000Z [(H)ERROR] tid(0x7f4158e84700) | :OE_ENCLAVE_ABORTING [/home/prp/sgx-lkl/openenclave/host/calls.c:oe_call_enclave_function_by_table_id:91]
[   SGX-LKL  ] ethread (3: 19) [   SGX-LKL  ] FAIL: sgxlkl_ethread_init() failed (id=3 result=19 (OE_ENCLAVE_ABORTING))

I wonder if this is caused by #644 because signals are not delivered to the correct thread. (I suspect that the lkl_bug above (bad count while changing owner (cpu.count=2 cpu.owner=139844147022336 owner=139844147019296)) is a consequence of the signal handler trying to do a state dump to a file.)

On the other hand, this problem only seems to be appearing in hw mode and not in sw mode. It also seems to be more prevalent for multi-threaded tests...?

It also doesn't occur if JIT is turned off.
@prp prp added bug area: sgx-lkl Core SGX-LKL functionality p1 Medium priority labels Jul 19, 2020
@prp prp added this to the Milestone 1 milestone Jul 19, 2020
@prp prp mentioned this issue Jul 19, 2020
Open
@prp prp changed the title Some Java DaCapo benchmarks fail with unhandled SIGSEGVs Some Java DaCapo benchmarks fail with unhandled SIGSEGVs in hw mode Jul 19, 2020
@prp
Copy link
Member Author

prp commented Jul 20, 2020 •
edited

With LKL syscall tracing (ignoring clock_gettime and futex calls):

[[ LKL SYSC ]] [tid=47 ] openat 56      (4294967196, /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so, 0, 438) = 20
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578896, 2, 0, 0, 0) = 1087
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 8941576, 0, 0, 0, 0) = 8941576
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578896, 2, 0, 0, 0) = 1087
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 0, 1, 0, 0, 0) = 8942663
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 9584, 0, 0, 0, 0) = 9584
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578784, 2, 0, 0, 0) = 12711
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 8941832, 0, 0, 0, 0) = 8941832
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578896, 2, 0, 0, 0) = 1087
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 0, 1, 0, 0, 0) = 8942919
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 21272, 0, 0, 0, 0) = 21272
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578784, 2, 0, 0, 0) = 8805
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 8941896, 0, 0, 0, 0) = 8941896
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578896, 2, 0, 0, 0) = 1087
[[ LKL SYSC ]] [tid=47 ] read   63      (20, 140600208234512, 1024, 0, 0, 0) = 385
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 0, 1, 0, 0, 0) = 8943368
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 8941344, 0, 0, 0, 0) = 8941344
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578784, 2, 0, 0, 0) = 1250
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 8943368, 0, 0, 0, 0) = 8943368
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, 64, 0, 0, 0, 0) = 64
[[ LKL SYSC ]] [tid=47 ] readv  65      (20, 140601075578976, 2, 0, 0, 0) = 1079
[[ LKL SYSC ]] [tid=47 ] lseek  62      (20, -687, 1, 0, 0, 0) = 456
[[ LKL SYSC ]] [tid=47 ] close  57      (20, 0, 0, 0, 0, 0) = 0
[[ LKL SYSC ]] [tid=47 ] openat 56      (4294967196, /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so, 524288, 0) = 20
[[ LKL SYSC ]] [tid=47 ] fcntl  25      (20, 2, 1, 0, 0, 0) = 0
[[ LKL SYSC ]] [tid=47 ] fstat  80      (20, 140601075577632, 0, 0, 0, 0) = 0
[[ LKL SYSC ]] [tid=47 ] close  57      (20, 0, 0, 0, 0, 0) = 0

Thread 11 "ENCLAVE" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc9ffb700 (LWP 26646)]
0x00007fe03865f3ef in ?? ()

@prp prp mentioned this issue Jul 23, 2020
Closed
@prp
Copy link
Member Author

prp commented Jul 27, 2020

This appears to be caused by the fact that the in-enclave signal handler does not report the faulting address of a page fault correctly to the JVM. With SGX1, there does not seem to be a way to obtain the address that caused the page fault.

@prp prp mentioned this issue Jul 27, 2020
Open
3 tasks
@davidchisnall davidchisnall added the needs-triage Bug does not yet have a priority assigned label Jul 28, 2020
@SeanTAllen
Copy link
Contributor

During triaging, it was decided to close this as a duplicate of #680.

@SeanTAllen SeanTAllen removed the needs-triage Bug does not yet have a priority assigned label Jul 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: sgx-lkl Core SGX-LKL functionality bug p1 Medium priority
Projects
None yet
Milestone
Milestone 1
Development

Successfully merging a pull request may close this issue.

Add Java DaCapo benchmark sample + provide fixes lsds/sgx-lkl
3 participants
@SeanTAllen @davidchisnall @prp