loading goenable crashes bash

[jakewarren]

When I run enable -f ./goenable.so goenable it crashes the bash process. I have no idea how to troubleshoot it further. Let me know if I can collect any logs or info that would be helpful.

Bash version

❯ bash --version | head -1
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

OS info

❯ whatversionami 
Linux jwarren 4.8.0-53-generic #56~16.04.1-Ubuntu SMP Tue May 16 01:18:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available.
Distributor ID:	LinuxMint
Description:	Linux Mint 18.3 Sylvia
Release:	18.3
Codename:	sylvia

PRETTY_NAME="Linux Mint 18.3"
UBUNTU_CODENAME=xenial ( 16.04 LTS )

[JohnStarich]

Thanks for the issue! I can reproduce reproduce with Bash 4.3 on macOS, but not on 4.4.

My theory is this could be an issue with the version of Bash goenable was compiled against. Can you test if this happens with Bash 4.4 or Bash 5?

I'll try recompiling against a few other versions to see if that fixes it.

[JohnStarich]

After some digging, I think the problem is in the Bash 4.3 / 4.4 boundary. It seems the Bash loadable's name is assumed to be non-NULL in Bash 4.3 and earlier, and goenable intentionally defers this until the Go runtime is started. I'll start on a fix soon! 😄

[JohnStarich]

@jakewarren Can you please try this prerelease and let me know if it solves your issue? https://github.com/JohnStarich/goenable/releases/tag/untagged-82eebf0d61329d0eca4a

I've tested lightly on macOS with Bash 3.2, 4.0-4.4, and 5.0 plus ubuntu with Bash 4.3 and 4.4.

[jakewarren]

@JohnStarich thanks for looking into it! I tested the prelease and it definitely made progress. I'm able to successfully load goenable and get help output. However when I try to load the pow example it crashes bash.

It spit out what looked a Go panic stack trace but it exited so quickly that I couldn't copy it. Even running it while redirecting the output with &> doesn't seem to work. I was able to find the bash stack trace but not sure if it'll be helpful.

Sorry I can help out more, don't mean to create more work for ya. :/

Stack trace

Stacktrace:
#0  0x00007ff234ab6144 in runtime.raise () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a9d1cb in runtime.dieFromSignal () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x00007ffd00000006 in ?? ()
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
StacktraceTop:
runtime.raise () from /tmp/goenable-Linux-x86_64.so
runtime.dieFromSignal () from /tmp/goenable-Linux-x86_64.so
?? ()
?? ()
ThreadStacktrace:
.
Thread 7 (Thread 0x7ff230532700 (LWP 5622)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86a8b in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x000000c000076140 in ?? ()
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 6 (Thread 0x7ff231534700 (LWP 5619)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86a8b in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x000000c00004a840 in ?? ()
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 5 (Thread 0x7ff22fd31700 (LWP 5621)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86a8b in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x00007ff234e5aaf8 in runtime.newmHandoff () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 4 (Thread 0x7ff230d33700 (LWP 5620)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86a8b in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x000000c00004abc0 in ?? ()
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 3 (Thread 0x7ff234746700 (LWP 5616)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86a8b in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x00007ff234e3f020 in runtime.m0 () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 2 (Thread 0x7ff231d35700 (LWP 5618)):
#0  0x00007ff234ab6693 in runtime.futex () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a86b00 in runtime.futexsleep () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x00007ff234e3e910 in runtime.sched () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#3  0x0000000000000080 in ?? ()
No symbol table info available.
#4  0x00007ff231d34db0 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
.
Thread 1 (Thread 0x7ff236360700 (LWP 3699)):
#0  0x00007ff234ab6144 in runtime.raise () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#1  0x00007ff234a9d1cb in runtime.dieFromSignal () from /tmp/goenable-Linux-x86_64.so
No symbol table info available.
#2  0x00007ffd00000006 in ?? ()
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
Title: bash crashed with SIGABRT in runtime.raise()

[JohnStarich]

Nice! 😄 Glad we got past that bit. I'll cut a release for that fix then!

You could try running bash as a subprocess, that way the output doesn't disappear when the subprocess exits. Can you run the following example and grab the output? (copy-paste may be required for this one, rather than file redirection with &>)

~ $ echo "I'm logged in!"
~ $ bash
bash-4.3 $ echo "Now I'm in the bash subshell"
bash-4.3 $ enable -f ./goenable.so goenable && make plugins && goenable load ./out/hello output

Perhaps the below trace looks similar to the Go stack trace you saw? This could be an issue I discovered which affects Linux systems but not macOS. I'm tracking this golang issue, which may help us figure out what went wrong: golang/go#18123 (comment)

No worries, I really appreciate the feedback! 👍

Go trace

+ goenable load ./out/pow output
fatal error: runtime: no plugin module data

goroutine 17 [running, locked to thread]:
runtime.throw(0x7fc8162505d6, 0x1e)
        /usr/local/go/src/runtime/panic.go:608 +0x74 fp=0xc000095738 sp=0xc000095708 pc=0x7fc8161b6bd4
plugin.lastmoduleinit(0xc000010240, 0xc00000e028, 0x1714008, 0xc, 0x7fc8165884e0)
        /usr/local/go/src/runtime/plugin.go:20 +0xb51 fp=0xc000095858 sp=0xc000095738 pc=0x7fc8161b7fd1
plugin.open(0xc000018044, 0x9, 0xc, 0x0, 0x0)
        /usr/local/go/src/plugin/plugin_dlopen.go:87 +0x3be fp=0xc000095b10 sp=0xc000095858 pc=0x7fc81624565e
plugin.Open(0xc000018044, 0x9, 0xc000018080, 0xc, 0x7fc8165884e0)
        /usr/local/go/src/plugin/plugin.go:32 +0x37 fp=0xc000095b48 sp=0xc000095b10 pc=0x7fc816244e57
main.run(0xc0000120c0, 0x3, 0x2, 0x10, 0x7fc817576000, 0x0, 0xc000018050)
        /src/hooks.go:191 +0x281 fp=0xc000095d50 sp=0xc000095b48 pc=0x7fc8162494f1
main.Run(0xc0000120c0, 0x3, 0x4, 0x2)
        /src/hooks.go:70 +0x5c fp=0xc000095e18 sp=0xc000095d50 pc=0x7fc8162488fc
main.goenable_builtin(0x170ff28, 0x7fc8164bf8d0)
        /src/bash.go:32 +0x153 fp=0xc000095e90 sp=0xc000095e18 pc=0x7fc81624aaf3
main._cgoexpwrap_0b007f78c50b_goenable_builtin(0x170fb08, 0x4cb89d)
        _cgo_gotypes.go:100 +0x2d fp=0xc000095eb0 sp=0xc000095e90 pc=0x7fc81624a67d
runtime.call32(0x0, 0x7ffd3651c2b0, 0x7ffd3651c340, 0x10)
        /usr/local/go/src/runtime/asm_amd64.s:522 +0x3d fp=0xc000095ee0 sp=0xc000095eb0 pc=0x7fc8161e04ed
runtime.cgocallbackg1(0x0)
        /usr/local/go/src/runtime/cgocall.go:313 +0x19f fp=0xc000095f58 sp=0xc000095ee0 pc=0x7fc81618fd6f
runtime.cgocallbackg(0x0)
        /usr/local/go/src/runtime/cgocall.go:191 +0xdc fp=0xc000095fc0 sp=0xc000095f58 pc=0x7fc81618fb2c
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/asm_amd64.s:775 +0x9a fp=0xc000095fe0 sp=0xc000095fc0 pc=0x7fc8161e1afa
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x7fc8161e2251

[jakewarren]

Yup looks like we get the same go trace.

Go trace

❯ goenable load ./out/hello output
fatal error: runtime: no plugin module data

goroutine 17 [running, locked to thread]:
runtime.throw(0x7fa727198eb4, 0x1e)
  /usr/local/go/src/runtime/panic.go:608 +0x74 fp=0xc00005c738 sp=0xc00005c708 pc=0x7fa7270ff184
plugin.lastmoduleinit(0xc0000901e0, 0xc00000e028, 0x32b5808, 0x17, 0x7fa7274d14e0)
  /usr/local/go/src/runtime/plugin.go:20 +0xb51 fp=0xc00005c858 sp=0xc00005c738 pc=0x7fa727100581
plugin.open(0xc000016170, 0xb, 0x17, 0x0, 0x0)
  /usr/local/go/src/plugin/plugin_dlopen.go:87 +0x3be fp=0xc00005cb10 sp=0xc00005c858 pc=0x7fa72718dc0e
plugin.Open(0xc000016170, 0xb, 0xc000018480, 0x17, 0x7fa7274d14e0)
  /usr/local/go/src/plugin/plugin.go:32 +0x37 fp=0xc00005cb48 sp=0xc00005cb10 pc=0x7fa72718d407
main.run(0xc0000280c0, 0x3, 0x2, 0x10, 0x7fa728991000, 0x0, 0xc000016180)
  /ext-go/1/src/github.com/JohnStarich/goenable/hooks.go:191 +0x281 fp=0xc00005cd50 sp=0xc00005cb48 pc=0x7fa727191d21
main.Run(0xc0000280c0, 0x3, 0x4, 0x2)
  /ext-go/1/src/github.com/JohnStarich/goenable/hooks.go:70 +0x5c fp=0xc00005ce18 sp=0xc00005cd50 pc=0x7fa72719112c
main.goenable_builtin(0x2a1a288, 0x0)
  /ext-go/1/src/github.com/JohnStarich/goenable/bash.go:32 +0x153 fp=0xc00005ce90 sp=0xc00005ce18 pc=0x7fa727193323
main._cgoexpwrap_727e109c28c2_goenable_builtin(0x2a1ac88, 0x2a0b1e8)
  _cgo_gotypes.go:100 +0x2d fp=0xc00005ceb0 sp=0xc00005ce90 pc=0x7fa727192ead
runtime.call32(0x0, 0x7ffc29bfa3a0, 0x7ffc29bfa430, 0x10)
  /usr/local/go/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00005cee0 sp=0xc00005ceb0 pc=0x7fa727128a9d
runtime.cgocallbackg1(0x0)
  /usr/local/go/src/runtime/cgocall.go:313 +0x19f fp=0xc00005cf58 sp=0xc00005cee0 pc=0x7fa7270d831f
runtime.cgocallbackg(0x0)
  /usr/local/go/src/runtime/cgocall.go:191 +0xdc fp=0xc00005cfc0 sp=0xc00005cf58 pc=0x7fa7270d80dc
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
  /usr/local/go/src/runtime/asm_amd64.s:775 +0x9a fp=0xc00005cfe0 sp=0xc00005cfc0 pc=0x7fa72712a0aa
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x7fa72712a801

[JohnStarich]

Dang, ok. I'll keep pinging this issue over on golang to get this issue looked at. Right now it's on the 1.11 milestone, but it's already been delayed a few times so this could be a while.

In the meantime, I'm sure you're anxious to get this working, so you could fork goenable and replace its innards with your own as a sort of "template" project. This would let you do enable -f ./mygo.so mygo and you can register whichever commands directly, rather than loading them as plugins. Also feel free to open new issues if you have questions about how goenable works! 😄

Once a fix for Linux arrives, I'll ping back here and let you know!

[JohnStarich]

@jakewarren According to the maintainers, Go plugins are no longer being worked on.. so I guess this project needs to pivot somehow if it's going to be used on Linux. 😕

[JohnStarich]

It should be easy enough to dynamically load an .so by using the usual C tools, question is whether this will wreak havoc with one or multiple Go runtimes. 🤔