ldex compiled for SunOS5/SPARC, Linux/armv7l with gcc takes segmentation fault in GC shortly after starting full.sysout

[nbriggs]

The "link" pointer is pointing at unallocated memory. The same code is executed on little-endian machines, so I'm suspecting an endian related issue in one of the helpful macros used in this code. The problem is reproducible.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
gcmapunscan () at ../src/gcmain3.c:366
366           offset = ((struct htcoll *)link)->free_ptr;
(gdb) where
#0  gcmapunscan () at ../src/gcmain3.c:366
Interlisp/maiko#1  0x0005fc68 in dogc01 () at ../src/gcr.c:124
Interlisp/maiko#2  0x0005fd58 in doreclaim () at ../src/gcr.c:139
Interlisp/maiko#3  0x0003ee8c in OP_subrcall (subr_no=<optimized out>, argnum=0) at ../src/subr.c:392
Interlisp/maiko#4  0x000324d4 in dispatch () at ../src/xc.c:741
Interlisp/medley#23  0x0004b1ec in main (argc=<optimized out>, argv=0xffbff8a4) at ../src/main.c:638
(gdb) print link
$1 = (LispPTR *) 0xfc65c000

[nbriggs]

Compiling only gcmain3.c with -O0 instead of -O2 "fixes" the problem. It's also the case that adding a printf of "link" or referencing it in other ways causes the code to behave correctly.

The compiler used was:

gcc --version
gcc (GCC) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.

[nbriggs]

The code appears to work when compiled with the Solaris Developer Studio 12.6 compiler with optimization on. There are a few adjustments necessary to include files to make that compiler happy (need to include arith.h which defines the macro N_ARITH_SWITCH anywhere that includes "my.h" since functions defined in "my.h" use the macro even though they themselves aren't used... implicit function definition leads to an undefined reference)

[nbriggs]

The same issue happens with gcc compiled code on armv7l architecture.

debian@beaglebone:~$ gcc --version
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[...]
(gdb) run full.sysout
Starting program: /home/debian/maiko/linux.armv7l/ldex full.sysout
Failed to find UNIXCOMM file handles; no processes

Program received signal SIGSEGV, Segmentation fault.
gcmapunscan () at ../src/gcmain3.c:365
365	      offset = ((struct htcoll *)link)->free_ptr;
(gdb) where
#0  gcmapunscan () at ../src/gcmain3.c:365
Interlisp/maiko#1  0x00431d2c in dogc01 () at ../src/gcr.c:119
Interlisp/maiko#2  0x00431dc2 in doreclaim () at ../src/gcr.c:134
Interlisp/maiko#3  0x0041a90e in OP_subrcall (subr_no=<optimized out>, argnum=<optimized out>) at ../src/subr.c:387
Interlisp/maiko#4  0x0041271c in dispatch () at ../src/xc.c:736
Interlisp/medley#23  0x00405eb6 in main (argc=<optimized out>, argv=0xbefffc44) at ../src/main.c:633
(gdb) 

[nbriggs]

Adding -fno-strict-aliasing to the compilation of gcmain3.c appears to prevent the error, at least initially.

[pbaur]

I had a very quick look at this issue and did some tiny experiment. It does not look to me like the garbage collector is the culprit, but somewhere else memory gets corrupted. As I am completely new to the code base: does anybody here know how memory is supposed to be managed or is there any conceptual description anywhere? Or is the actual source code all we have? Just asking before I go down the rabbit hole...

[nbriggs]

Hi -- Get the latest update to source (or, more specifically, the makefiles) -- see commit cd4c59dbb3114e4d26260705cb3f10e7b4b67d31 the problem with the segfault in the GC code is a result of (mostly with gcc) not having -fno-strict-aliasing -- the code is very sloppy about pointer aliasing because it's running a virtual machine where memory is just an array of 16-bit words... except when it isn't. The authors defined various C structures, for ease of field access over multiple 16-bit words, and the compiler mustn't make *any* assumptions about about there being no aliasing between pointers of arbitrary type. This most notably shows up in the GC because it's trying to touch everything... If you, for example, toss in a printf to track things that are GC'd... the segfault goes away.

…

-- Nck Briggs

On Aug 23, 2020, at 6:24 PM, pbaur ***@***.***> wrote: I had a very quick look at this issue and did some tiny experiment. It does not look to me like the garbage collector is the culprit, but somewhere else memory gets corrupted. As I am completely new to the code base: does anybody here know how memory is supposed to be managed or is there any conceptual description anywhere? Or is the actual source code all we have? Just asking before I go down the rabbit hole... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/Interlisp/maiko/issues/3#issuecomment-678857821>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6DAWLMA3VYKVLB633XD4DSCG6N5ANCNFSM4PG7HZ4A>.

[nbriggs]

There's the original Interlisp-D implementation of the GC, and this C code is an implementation of both some VM opcodes that were microcoded on the D-machine and, at a slightly higher level, some Lisp code that used those opcodes. I don't think the Lisp code is up on github yet.

[pbaur]

I took the latest sources including the -fno-strict-aliasing, also tried compiling without optimization, both didn't help in my case. When I monitor the GC code, I still get pointers to nirvana...I tried the full.sysout (from Dropbox) as well as the xfull.sysout from the medley repo---same story with both. Thus, I'll start hunting things down, will likely take me a good while, getting an understanding of what's supposed to be and finally figuring why it is not.

Thank you very much for the quick response, explanation and hints. Being completely new to the code base, and apart from some historic reading on the architecture, new to the D-architecture and Interlisp-D; I have a little catching up to do. And while I am at it: lots of thanks for making the code available, I have always been intrigued by the machine as well as the software architecture of the D-Line, now I finally get a chance to dig into both. Looking forward to see this restoration project prosper, and happy to help (once I am at a point of being of any help, that is).

Last but not least, my apologies for not introducing myself properly, I wasn't aware my profile is nameless: I'm Patrick Baur.

[nbriggs]

Hi Patrick -- what OS and architecture are you compiling for, and with which compiler/version? In case we overlap: I've been building on MacOS 10.11 with clang 8.00 (standard Apple release) 32- and 64-bit, FreeBSD with gcc-9.3.0 and clang versions 8, 9, and 12, 32-bit only, Ubuntu 20.04.1 with clang 10.0 (64-bit), Solaris 11.4 on SPARC T1 Niagara with the Solaris 12.6 Studio compiler (mostly 32-bit), Solaris 11.4 on x86_64 but compiling 32-bit mode with clang 6.0.0, and most recently a Beaglebone Black (armv7l) running Debian 9.9, gcc (which I don't have running at the moment so I don't know which version).

[pbaur]

Hi Nick. I started out on my desktop with the precompiled version from the medley repo: AMD Ryzen 7, Fedora 32 (virtualized). Works like a charm, thus didn't yet go into recompiling it (which I will do soon to get a baseline to compare the Rpi to). That one is running gcc 10.2.1, current Fedora repo version.

On the Raspberry, I got a 4B/4GB (Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux) with gcc (Raspbian 8.3.0-6+rpi1) 8.3.0 and clang version 7.0.1-8+rpi3 (tags/RELEASE_701/final.

I might recover a Mac as well, would be current OS and compiler suite of theirs, if they still support the model.

[nbriggs]

OK. Not sure that Larry recompiled the versions in the repo after I had added the -fno-strict-aliasing, it may have been compiled with reduced optimization though. I'm afraid I don't have an RPi, so no joy there. Generally, I've been happier with the results from clang over gcc. With a currently supported Mac you'd be able to install Catalina and the latest XCode. If it's a Mac that's older you'd be able to install MacOS El Capitan (10.11.x) -- my everyday Mac is from 2008 and can't get past 10.11.x, but others are running Catalina (10.15.x).

[pbaur]

I just recompiled your newest maiko on my desktop, and it appears to be working fine, at least for some five minutes. That is with gcc (against your makefile which would favour clang, just didn't have it installed, thus took gcc for a spin first). Once it looks like it is working properly, I'll try clang.
I will see to get a Mac up, too. From my quick glimpse at the code it looks like i386 stuff is separate from the rest of the pack, thus might be that the non-Intel code got less love in the past. Does it work for you on your SPARCs?

[nbriggs]

The D-machines that this is emulating are big-endian, and the C code was originally running on Motorola 68020 Sun3 systems and then and on SPARC (both big-endian). The port to little-endian came later. Initially it was done on the Sun i386 boxes, and there is special dispensation beyond the generic little-endianness because specifically there the compiler behaved differently.
Since then, most of the hardware has been little-endian (and almost exclusively x86/x86_64), but I'm still finding bugs due to missing little-endian work (e.g., the error in FBITMAPBIT, issue Interlisp/maiko#13 which is a day-0 problem)

p.s., not my SPARC, but someone is kindly providing access to it.

[pbaur]

Actually, I was too forward. While it does not crash as such (drop you in the command monitor due to segfault), with optimization it hangs (does not accept any input anymore, but still using its 100% CPU---not sure whether it was still kinda happy or went berserk). As you mentioned, appears that the repo version was indeed compiled with reduced optimization. Taking optimiziation out, it looks fine, fingers crossed.
I can imagine that with the endianness change plus the address and word size changes, there might be some rough spots that were overlooked (I saw the twiddlings and maskings all over the place). That's a starting point. I am reading the VM specification, hopefully getting a bit more of an insight into how things are supposed to work memory-wise. I'll try to do an audit of all the memory mingling (endianness, address and word sizes), might be more fruitful than trying to find the needle(s) in the burning haystack. If I can get comfortable with the code, maybe there's ways to consolidate things and do all the architecture changes in one place consistently.

[nbriggs]

It always uses 100% of a cpu -- that's not a surprise since the idle loop and timers are within the emulated code. Normally you just click within a window and it will take the focus. I have seen very few cases of it locking up in a loop -- though it will drop into its own debugger, uraid, when things go south.

If you're looking at memory access, there are access macros in lsptypes.h for things like

#define GETBYTE(base) (* (unsigned char *) (3^(UNSIGNED)(base)))
  /* GETBASEWORD only works if base points to a 32-bit bounday */
#define GETBASEWORD(base, offset) GETWORDBASEWORD((base),(offset))
#define GETWORDBASEWORD(base, offset) (* (DLword *) (2^(UNSIGNED)((base)+(offset))))
#define GETWORD(base) (* (DLword *) (2^(UNSIGNED)(base)))
#define WORDPTR(base) ((DLword *)(2^(UNSIGNED)(base)))
#define BYTEPTR(base) ((char *) (3^(UNSIGNED)(base)))

and so on. There is also a lot of conditional code for things like BIGVM, BIGATOMS, which affects the sizes of objects that are referenced out of the Lisp virtual memory. FYI, these days it's all BIGVM and BIGATOMS.

As I mentioned, the code is very sloppy with regard to the sizes of objects it's accessing. I had quite a time getting it to run when compiled LP64, and you'll probably find things in there that I missed, though this problem in particular happens in ILP32 mode too.

[nbriggs]

This may not be the right place for this wide ranging discussion, but...
(1) events: there are certainly external events (keyboard events, arrival of network packets) and then the timers. As far as I remember, timer events run off the timer process, which is one of the basic threads that is always running. Things like the blinking cursor are also handled in Lisp.
(2) One can add "user" subrs that let you attach a piece of C code to implement a Lisp function. That's all already in the Lisp compilers (there are two of them)
(3) The emulator code can be compiled for different versions of Interlisp -- that generates a bunch of ifdefs through the source. Those things are fixed in history by the presence of old sysouts and old compiled code.
(4) If the internal sizes of objects within the lisp code change, you have to be able to migrate forward. This gets really complicated, because, remember, the C code implements the virtual machine but contains NONE of the system itself -- everything starts from the previous sysout. When you start a sysout it contains the current PC and the context that was left when the sysout was written... the VM starts it up right where it left off with all the state it previously had. If one is making changes one sometimes has to go through the MAKEINIT process, where one sysout (running on a specially modified VM emulator) creates a new static sysout that can afterwards be started with the new emulator. I suppose you could think of it like the birth of Athena ;-)
(5) Sysouts do not contain memory that is not in use. Pages are allocated to a datatype as required (includes cons cells). Only active pages are written out.

[pbaur]

Agreed, I already felt bad myself for usurping this issue's thread. Thank you for replying nonetheless, valuable insights, much appreciated!
For amd64, the clang appears to be working for me with -O2. For arm, none of gcc/clang/tcc work reliably. I have a reproducible arm case:

  0 :    0x1455e : IL:\UNWIND.UFN
  1 :    0x1444a : IL:\GETFBB
  2 :    0x143cc : IL:\READACFONTFILE
  3 :    0x143a4 : SI:*UNWIND-PROTECT*
  4 :    0x14376 : IL:\READDISPLAYFONTFILE
  5 :    0x1434e : IL:\CREATE-REAL-CHARSET.DISPLAY
  6 :    0x14326 : IL:\CREATECHARSET.DISPLAY
  7 :    0x14300 : IL:APPLY
  8 :    0x142e0 : IL:\CREATECHARSET
  9 :    0x142c4 : IL:\CREATEDISPLAYFONT
 10 :    0x14298 : IL:FONTCREATE
 11 :    0x1426a : IL:\EVALFORM
 12 :    0x1424c : IL:FAULTEVAL
 13 :    0x1422e : IL:\EVALFORM
 14 :    0x1421a : IL:EVAL
 15 :    0x141f6 : IL:EVAL-INPUT
 16 :    0x14184 : IL:DO-EVENT
 17 :    0x1414e : XCL:EXECA0001A0002
 18 :    0x14116 : XCL:EXECA0001
 19 :    0x14070 : IL:\DO.PROGV
 20 :    0x13ff2 : XCL:EXEC
 21 :    0x13fc6 : IL:\EVALFORM
 22 :    0x13fac : CL:PROGN
 23 :    0x13f8c : IL:\EVALFORM
 24 :    0x13f3e : IL:\MAKE.PROCESS0
 25 :    0x11802 : CL:T

Unlike the Lisp stack, the C stack appears to get completely obliterated. I read the VM specification, which is like the ham in the sandwich; I will try to get somewhat acquainted with the maiko code, but otherwise wait for the Lisp sources to be released, it is kinda hard to figure what's going wrong if you don't know what right would be to start with: (FONTCREATE 'MODERN 18 'BOLD) with DISPLAYFONTDIRECTORIES set and pointing to valid files crashes this way. It does not crash if I set DISPLAYFONTDIRECTORIES to NIL or try to create font there's no trace of, like (FONTCREATE 'FOO 10 'STANDARD)...

[masinter]

I'd look for a byteswap problem in the UNWIND opcode (which, I confess, I have no idea what it is supposed to do)

[nbriggs]

I can reproduce the FONTCREATE crash with the current code compiled with "gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516" on "Linux beaglebone 4.14.108-ti-r113 #1 SMP PREEMPT Wed Jul 31 00:01:10 UTC 2019 armv7l GNU/Linux" (armv7l-unknown-linux-gnueabihf) -- I think this is a different problem from the pointer aliasing problem in the garbage collector.

[pbaur]

I didn't spot anything on UNWIND-PROTECT in any of the Interlisp documentation, thus my guess is that it was introduced with the addition of Common Lisp. UNWIND-PROTECT is akin to C's setjump/longjump in that you remember the current state and try something. If all goes well, continue, if not, restore the remembered state and pretend nothing ever happened. As such, it implies some dark stack magic. My understanding is that the VM is designed to be a bookkeeper, mainly storing data and providing simple operations on it, no complex upper level logic; I am thus not sure why UNWIND ended up in the VM and was not implemented on the Lisp side, but I assume there were good reasons for that (e.g. performance or lacking operation).
The FONTCREATE case is an unfortunate one: for one, why is it triggering an UNWIND? It should not fail to read and process the font file in the first place (it does not on the 64bit variant, thus the file appears to be correct as well as the Lisp code reading and processing it). As it does trigger it, the question is now whether the culprit is the reading/processing part corrupting things, or the UNWIND, and if the latter, is the problem with remembering the state correctly or with restoring it.
I saw that you released the Lisp side of the code (thank you!), I will thus start looking into pinning that problem down---at least once I get somewhat more comfortable with snooping around, looking at bits and bytes, debugging/tracing/stepping through things, keeping an eye on both parties.
On a positive note---please excuse my French---I got fed up with Raspian's brittleness and switched to Ubuntu Mate (I would have preferred BSD, but they're still pending the support for the new USB chipset). As I was already on it, I took the 64bit variant for a spin, and---so far---it works smoothly with gcc and all optimizations turned on (-O2 and even -Ofast). I am not sure what that does tell us exactly, but looks like the 64bit variant either does something right the 32bit doesn't, or it is just lucky that the wrongdoing doesn't have any impact (i.e. is in the "unused" side of the word).

[pbaur]

From ADDARITH:

(RPAQQ MASK0WORD1'S 32767)
(RPAQQ MASK1WORD0'S 32768)
(RPAQQ MASKWORD1'S 65535)
(RPAQQ MASKHALFWORD1'S 255)
(RPAQQ BITSPERHALFWORD 8)

I don't know where this all is having effects, but looks like the Lisp side has its own idea about words, word sizes, etc.

[masinter]

We maintained Interlisp-10 (and Interlisp-MAXC) through years where those numbers were variables (or would appear in (CONSTANT BITSPERHALFWORD) macros.
maiko implements a virtual machine with a limited address space with memory aligned with "big-endian" 16 bit words. Any "little-endian" implementation has to swap bytes like crazy.

A general warning probably is in order for the LL- and A- files that they might look like they're in ordinary Interlisp, but there are (mainly not yet written) rules about reading and compiling.

The emulated address space changed twice by removing bits from CDR-coding.

[nbriggs]

As Larry says, the Lisp code running on the virtual machine is working with 16-bit big-endian words. Some things are coded (in Lisp) in multiple 16-bit words. The VM accesses some Lisp structures directly, and has to be aware of the possibility of an endianness mismatch between the Lisp view and the native machine view, therefore the access macros.

Layer on top of this the fact that the original code was implemented on an ILP32 (int/long/pointer all 32-bit) system, and then ported to a more modern ILP32 system (MacOS X) and then to an LP64 system (again MacOS X). Along the way the default changed from unsigned characters to signed characters (which we now force back again with -funsigned-char) which caused quite a few bugs, and there was a lot of sloppiness in the original code with regard to whether native pointers were declared as such -- e.g., conversion from a "lisp pointer" (an offset from 0 in the Lisp memory) to a native pointer represented as sometimes an integer (and variously int or unsigned) or a declared pointer (e.g., unsigned short * or sometimes int *). With the ILP32 to LP64 transition I had to track down a bunch of issues where it depended on wraparound in 32-bit arithmetic. Now add the optimizations that the new compilers assume they can do because the C standard has changed with regard to pointer aliasing... and you start to see why it's difficult to get/keep this old code running. Oh, and there is no guarantee that the original code was correctly ported to deal with the endianness issues (witness the FBITMAPBIT fix I made just the other day)

[masinter]

The notes about word size constants and other things belong in the documentation. The README is fine unless it gets too long.
The web site or somewhere it links to should note the Pi configuration preferred (so I know what to get when I order a Pi or beagleboard black or whatever.)

Are there any new makefile "parts" (cf #25 )?

[pbaur]

I tried to have another look at the issue but ran into some problem there: loading FONT from sources drops you into an endless loop complaining that \CHARCODEP is an undefined function. I'm not sure whether our sysout just doesn't match our sources or one has to do some special trickery to load such low level sources.

[nbriggs]

If you're poking around in lower-level code, and using sources rather than compiled code, you probably need to load EXPORTS.ALL first.

[nbriggs]

@pbaur -- Could you explain the reasoning that takes you to the FONT code (or any other Lisp code) rather than the VM emulator C code in search of this problem?

[pbaur]

Two reasons: (a) I would like to know what it wants to do before I try to figure why it fails to do it; (b) I'd like to isolate the problem as much as possible.
All I know so far is that somewhere between entering the UNWIND-PROTECT and the UNWIND being triggered the stack is corrupted. There's nothing jumping right at me from the C source that UNWIND is the culprit, less so as the UNWIND should not be triggered in the first place. Of course, I might be completely wrong there, whether I am just not seeing it or whatever else. That leaves a pretty broad scope for the faulty code location.
Working backwards from the segfault is not an option, as the stack is already crushed (even if it wasn't, there would be no guarantee that the culprit is still in the back trace, it might have come and gone). Looking through some 70K lines of C code trying to spot where things go wrong is somewhat off limits, as is stepping through hundreds of lines of Lisp code on the opcode level continuously checking stack and memory. Not that I would mind doing either, but I sadly don't have that amount of spare time.
If I can go through the lisp code and corner the portion that causes the problem, that would bring things down to a manageable scope. It might, of course, also happen that this ain't possible, that it is an unfortunate combination of mishaps that trigger the fault.
On EXPORTS.ALL: just tried that, the result is somewhat shifted, but basically the same: it now loops over SPREADAPPLY*is an undefined function`.
In my experience if one reloads function definitions with identical ones, the result should be the same, nothing but a costly no-op. Obviously I am missing something here, but what?

[nbriggs]

For stack issues, vs. the GC code issues, I've been compiling the C code with -DSTACKCHECK and it occasionally finds problems with the stack structure -- the code is the only "documentation" for the details of the Interlisp spaghetti stacks, and I haven't had a chance to work out what things should really look like.

[masinter]

This discussion is all over but it isn't clear what the problem is. For now, this is a Discussion.