Invalid CPU instructions generated by new dynarec

[269261]

Loading and executing one of the attached files (s4.zip) results either in SIGILL (invalid instruction in extra_buffer) or SIGSEGV.

For SIGILL-causing file (cpuinstr1.z64) the invalid instruction generation seems to be caused by emit_writeword with rt=-1 argument (it seems to be wrong since regname[rt] is used inside this function, and it should not receive negative index)

mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c

Lines 2309 to 2316 in f500eb5

	static void emit_writeword(int rt, intptr_t addr)
	{
	assert((intptr_t)addr-(intptr_t)out>=-2147483648LL&&(intptr_t)addr-(intptr_t)out<2147483647LL);
	assem_debug("movl %%%s,%llx",regname[rt],addr);
	output_byte(0x89);
	output_modrm(0,5,rt);
	output_w32(addr-(intptr_t)out-4); // Note: rip-relative in 64-bit mode
	}

It may be the reason of the SIGILL.

Debugging cpuinstr1.z64 execution:

$ gdb -ex run --args Bin/Release/RMG /tmp/cpuinstr1.z64
GNU gdb (Ubuntu 13.1-2ubuntu2) 13.1
...

Thread 14 "Thread::Emulati" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7fffa63fd6c0 (LWP 204015)]
0x00007fffcc9ee05e in g_dev () from /tmp/RMG/Bin/Release/Core/libmupen64plus.so
(gdb) disas /r $pc,$pc+20
Dump of assembler code from 0x7fffcc9ee05e to 0x7fffcc9ee072:
=> 0x00007fffcc9ee05e <g_dev+9441374>:	ce                 	(bad)
   0x00007fffcc9ee05f <g_dev+9441375>:	00 00              	add    BYTE PTR [rax],al
   0x00007fffcc9ee061 <g_dev+9441377>:	02 89 1d 98 00 00  	add    cl,BYTE PTR [rcx+0x981d]
   0x00007fffcc9ee067 <g_dev+9441383>:	02 50 51           	add    dl,BYTE PTR [rax+0x51]
   0x00007fffcc9ee06a <g_dev+9441386>:	48 83 c4 90        	add    rsp,0xffffffffffffff90
   0x00007fffcc9ee06e <g_dev+9441390>:	bf 98 05 00 a4     	mov    edi,0xa4000598
End of assembler dump.
(gdb) set $offset = $pc-g_dev.r4300.extra_memory
(gdb) b new_dyna_start
Breakpoint 1 at 0x7fffcc0bf70b
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /tmp/RMG/Bin/Release/RMG /tmp/cpuinstr1.z64
...

Thread 14 "Thread::Emulati" hit Breakpoint 1, 0x00007fffc80bf70b in new_dyna_start () from /tmp/RMG/Bin/Release/Core/libmupen64plus.so
(gdb) p/x $offset
$1 = 0x5e
(gdb) watch *(g_dev.r4300.extra_memory+$offset)
Hardware watchpoint 2: *(g_dev.r4300.extra_memory+$offset)
(gdb) c
Continuing.

Thread 14 "Thread::Emulati" hit Hardware watchpoint 2: *(g_dev.r4300.extra_memory+$offset)

Old value = 0 '\000'
New value = -50 '\316'
output_w32 (word=33554638) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:747
747	  out+=4;
(gdb) bt
#0  output_w32 (word=33554638) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:747
#1  0x00007fffc808690e in emit_writeword (rt=-1, addr=140736592798000) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:2315
#2  0x00007fffc8096d76 in do_readstub (n=0) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/new_dynarec.c:4817
#3  0x00007fffc80bd59c in new_recompile_block (addr=-1543503808) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/new_dynarec.c:11680
#4  0x00007fffc80bf723 in new_dyna_start () at /tmp/RMG/Bin/Release/Core/libmupen64plus.so
#5  0x0000000000000000 in  ()

When it comes to SIGSEGV-causing file, I don't know the reason. Please note that the similar issue may exist in related files too (e.g. assem_x86.c).

Test platform

• Ubuntu 23.04 (x86-64)
• RMG emulator (https://github.com/Rosalie241/RMG)
  • built from master branch
  • Release build, x86-64
  • custom flags added to mupen64plus-core Makefile:
    • CFLAGS += -O0 -g
    • -O0 and -g for better crash backtraces in Release build
• gdb 13.1 for debugging