Loader could not find fstat

[ValdikSS]

Trying to study the code to understand why fstat symbol could not be found in the wrapped libc. It seems to be wrapped, and wrapper generator complains if I explicitly add the definition as under #ifdef ANDROID in src/wrapped/wrappedlibc_private.h, but it still complains.

Do I understand correctly that the wrappers were written to a specific base userspace libraries (Steam's Ubuntu I guess?) and won't work if the library such as libc gets updated?
I'm running Debian 12 armhf with i386 libraries (also Debian 12, glibc 2.36-9+deb12u3).

$ cat test.c 
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/stat.h>

int main() {
  // int fstat(int fd, struct stat *statbuf);
  struct stat sb;
  int a = fstat(0, &sb);

  return 0;
}

$ i686-linux-gnu-gcc test.c

$ BOX86_DLSYM_ERROR=1 BOX86_LOG=2 box86 /tmp/a.out 
Debug level is 2
Shows details of dlopen / dlsym /dlclose : Yes
Dynarec for ARMv7, with extension: HALF FAST_MULT EDSP NEON VFPv4 SWP IDIVA PageSize:4096 Running on Cortex-A7 with 1 Cores
Params database has 32 entries
Box86 with Dynarec v0.3.4 nogit built on Jan 18 2024 07:41:17
Setting up canary (for Stack protector) at GS:0x14, value:ADEED100
Using default BOX86_LD_LIBRARY_PATH: ./:lib/:lib32/:x86/:i686/
Using default BOX86_PATH: ./:bin/
Counted 16 Env var
Looking for /tmp/a.out
Read 30 Section header
Read 11 Program header
Loading Sections Table String (idx = 29)
Loading SymTab Strings (idx = 28)
Loading SymTab (idx = 27)
Loading Dynamic (idx = 21)
The DT_INIT is at address 0x1000
The DT_FINI is at address 0x11d0
The DT_INIT_ARRAY is at address 0x3ee8
The DT_INIT_ARRAYSZ is 1
The DT_FINI_ARRAY is at address 0x3eec
The DT_FINI_ARRAYSZ is 1
The DT_VERNEED is at address 0x344
The DT_VERNEEDNUM is 1
Rel Table @0x384 (0x40/0x8)
PLT Table @0x3c4 (type=17 0x10/0x8)
The GOT.PLT Table is at address 0x3ff4
The GOT Table is at address 0x3fe0..0x3ff4
The PLT Table is at address 0x1020..0x1050
The .gnu.version is at address 0x334
The .text is at address 0x1060, and is 366 big
The .bss is at address 0x4010, and is 4 big
The .eh_frame section is at address 0x203c..0x2104
The .eh_frame_hdr section is at address 0x2008
Loading DynSym Strings (idx = 6)
Loading DynSym (idx = 5)
Adding "/tmp/a.out" as #0 in elf collection
Elf Addr(v/p)=(nil)/(nil) Memsize=0x4014 (align=0x1000)
Elf Stack Memsize=1048576 (align=16)
Elf TLS Memsize=0 (align=0)
Pre-allocated 0x4014 byte at 0x30000000 for /tmp/a.out
Delta of 0x30000000 (vaddr=(nil)) for Elf "/tmp/a.out" (0x4014 bytes)
8749|Allocate a new X86 Emu, with EIP=(nil) and Stack=0xb6cf0000/0xC0000
Setup X86 Emu
Mmaping 0x3d4 memory @0x30000000 for Elf "/tmp/a.out"
Mmaping 0x1e4 memory @0x30001000 for Elf "/tmp/a.out"
Mmaping 0x104 memory @0x30002000 for Elf "/tmp/a.out"
Allocating 0x2000 (0x12c) bytes @0x30003000, will read 0x128 @0x30003ee8 for Elf "/tmp/a.out"
Rename process to "a.out"
Program linked with GLIBC 2.34+
Calc stack size, based on 1 elf(s)
Stack is @0xb64f0000 size=0x800000 align=0x10
8749|Allocate a new X86 Emu, with EIP=(nil) and Stack=0xb64f0000/0x800000
Setup X86 Emu
8749|Free a X86 Emu (0x640a16c8)
Trying to add "libc.so.6" to maplib
Trying to load "libc.so.6"
Simplified name is "libc.so.6"
Using native(wrapped) libc.so.6
Trying to add "ld-linux.so.2" to maplib
Trying to load "ld-linux.so.2"
Simplified name is "ld-linux.so.2"
Using native(wrapped) ld-linux.so.2
Trying to add "libpthread.so.0" to maplib
Trying to load "libpthread.so.0"
Simplified name is "libpthread.so.0"
Using native(wrapped) libpthread.so.0
Trying to add "librt.so.1" to maplib
Trying to load "librt.so.1"
Simplified name is "librt.so.1"
Using native(wrapped) librt.so.1
Trying to add "libdl.so.2" to maplib
Trying to load "libdl.so.2"
Simplified name is "libdl.so.2"
Using native(wrapped) libdl.so.2
Trying to add "libm.so.6" to maplib
Trying to load "libm.so.6"
Simplified name is "libm.so.6"
Using native(wrapped) libm.so.6
Trying to add "libbsd.so.0" to maplib
Trying to load "libbsd.so.0"
Simplified name is "libbsd.so.0"
Using native(wrapped) libbsd.so.0
Created lib and added to maplib => success
Created lib and added to maplib => success
Created lib and added to maplib => success
Created lib and added to maplib => success
Created lib and added to maplib => success
Created lib and added to maplib => success
Created lib and added to maplib => success
And now export symbols / relocation for /tmp/a.out...
Applying 8 Relocation(s) for /tmp/a.out
Applying 2 PLT Relocation(s) for /tmp/a.out
PLT Resolver injected in plt.got at 0x30003ffc
Entry Point is 0x30001060
Start x86emu on Main
 ...  =>PltResolver: Addr=0x640a03c0, Slot=0 Return=0x30001087: elf is /tmp/a.out (VerSym=0x334)
            Apply Global R_386_JMP_SLOT 0x30004000 with sym=__libc_start_main(ver 2: __libc_start_main@GLIBC_2.34) (0x30001036 -> 0x40030040 / (noelf))
 return 0x00000001
8749|0x30001087: Calling my___libc_start_main (/tmp/a.out) (30001189, 00000001, B6CEFD9C...) =>Transfert to main(1, 0xb6cefd9c, 0xb6cefda4)=>0x30001189 from __libc_start_main
 ...  =>PltResolver: Addr=0x640a03c0, Slot=8 Return=0x300011b5: elf is /tmp/a.out (VerSym=0x334)
Error: PltResolver: Symbol fstat(ver 4: fstat@GLIBC_2.33) not found, cannot apply R_386_JMP_SLOT 0x30004004 (0x30001046) in /tmp/a.out
 return 0x30003FF4
 return 0x30003FF4
Emulation finished, EAX=805322740
Calling atexit registered functions
Calling atexit registered functions
Calling fini for all loaded elfs and unload native libs
Calling Fini[0] for /tmp/a.out 0x30001130
8749|0x30001163: Calling my___cxa_finalize (/tmp/a.out) (3000400C, 00000000, 00000000...) =>Calling atexit registered functions for 0x3000400c mask
 return 0x40030010
Calling Fini for /tmp/a.out 0x300011d0
Waiting for all threads to finish before unloading box86context
8749|Free a X86 Emu (0x6409f3b8)

[ValdikSS]

When running in fully emulated mode, it crashes with SIGSEGV
0x2a reloc type is R_386_IRELATIVE.

# BOX86_DLSYM_ERROR=1 BOX86_LOG=1 BOX86_PREFER_EMULATED=1 BOX86_EMULATED_LIBS=libc.so.6 BOX86_DYNAREC=0 BOX86_SHOWSEGV=1 BOX86_SHOWBT=1 box86 /tmp/a.out 
Debug level is 1
Dynarec is off
Shows details of dlopen / dlsym /dlclose : Yes
Show Segfault signal even if a signal handler is present
Show Backtrace for signals
Running on Cortex-A7 with 2 Cores
Params database has 32 entries
Box86 with Dynarec v0.3.4 nogit built on Jan 18 2024 07:41:17
Using default BOX86_LD_LIBRARY_PATH: ./:lib/:lib32/:x86/:i686/
BOX86 will force the used of emulated libs for libc.so.6 
Using default BOX86_PATH: ./:bin/
BOX86: Prefer Emulated libs
Counted 16 Env var
Looking for /tmp/a.out
Rename process to "a.out"
Using emulated /lib/i386-linux-gnu/libc.so.6
Redirecting overridden malloc function for /lib/i386-linux-gnu/libc.so.6
Using emulated /lib/i386-linux-gnu/ld-linux.so.2
Warning, don't know how to handle rel #2 type: 0x2a (unknown) (0x60432ba0) for 
Warning, don't know how to handle rel #92 type: 0x2a (unknown) (0x6021c844) for 
Warning, don't know how to handle rel #15 type: 0x2a (unknown) (0x6021d03c) for 
Warning, don't know how to handle rel #16 type: 0x2a (unknown) (0x6021d01c) for 
Warning, don't know how to handle rel #17 type: 0x2a (unknown) (0x6021d014) for 
Warning, don't know how to handle rel #18 type: 0x2a (unknown) (0x6021d004) for 
Warning: Global Symbol _ITM_deregisterTMCloneTable not found, cannot apply R_386_GLOB_DAT @0x30003fe0 ((nil)) in /tmp/a.out
Warning: Global Symbol _ITM_registerTMCloneTable not found, cannot apply R_386_GLOB_DAT @0x30003ff0 ((nil)) in /tmp/a.out
10673|SIGSEGV @0x62ae72dc (???(/tmp/a.out/0x62ae72dc)) (x86pc=0x230d0/???:"???", esp=0xb6cefd88, stack=0xb64f0000:0xb6cf0000 own=(nil) fp=(nil)), for accessing 0x230d0 (code=1/prot=0), db=(nil)((nil):(nil)/(nil):(nil)/???:clean, hash:0/0)
ESP-0x10:0x00000000 ESP-0x0c:0x00000000 ESP-0x08:0x00000000 ESP-0x04:0x00000000
ESP+0x00:0x40020010 ESP+0x04:0x00000001 ESP+0x08:0xb6cefd9c ESP+0x0c:0xb6cefda4
Native bactrace:
        /tmp/a.out() [0x629609c0]
        /lib/arm-linux-gnueabihf/libc.so.6(+0x2e1b0) [0xb6e791b0]
        /tmp/a.out() [0x62ae72dc]
        /tmp/a.out() [0x62969650]
        /tmp/a.out() [0x629697f8]
        /tmp/a.out() [0x6289b244]
        /tmp/a.out() [0x62936674]
        /tmp/a.out() [0x62936848]
        /tmp/a.out() [0x628899e8]
        /lib/arm-linux-gnueabihf/libc.so.6(+0x1e2da) [0xb6e692da]
        /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x5d) [0xb6e6938a]
10673|Double SIGSEGV (code=1, pc=0x62ae72dc, addr=0x230d0)!

And the similar issue with symbol loading when running in prefer-emulated mode:

# BOX86_DLSYM_ERROR=1 BOX86_LOG=1 BOX86_PREFER_EMULATED=1 box86 /tmp/a.out 
Debug level is 1
Shows details of dlopen / dlsym /dlclose : Yes
Dynarec for ARMv7, with extension: HALF FAST_MULT EDSP NEON VFPv4 SWP IDIVA PageSize:4096 Running on Cortex-A7 with 2 Cores
Params database has 32 entries
Box86 with Dynarec v0.3.4 nogit built on Jan 18 2024 07:41:17
Using default BOX86_LD_LIBRARY_PATH: ./:lib/:lib32/:x86/:i686/
Using default BOX86_PATH: ./:bin/
BOX86: Prefer Emulated libs
Counted 16 Env var
Looking for /tmp/a.out
Rename process to "a.out"
Using native(wrapped) libc.so.6
Using emulated /lib/i386-linux-gnu/ld-linux.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Using native(wrapped) libdl.so.2
Using native(wrapped) libm.so.6
Using emulated /lib/i386-linux-gnu/libbsd.so.0
Warning, don't know how to handle rel #2 type: 0x2a (unknown) (0x60032ba0) for 
Using emulated /lib/i386-linux-gnu/libmd.so.0
Using native(wrapped) libc.so.6
Error: Global Symbol __progname not found, cannot apply R_386_GLOB_DAT @0x60215fd8 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Warning: Global Symbol _ITM_deregisterTMCloneTable not found, cannot apply R_386_GLOB_DAT @0x60215fdc ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: Global Symbol printf not found, cannot apply R_386_GLOB_DAT @0x60215fe0 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: Global Symbol stderr not found, cannot apply R_386_GLOB_DAT @0x60215fe4 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: Global Symbol optind not found, cannot apply R_386_GLOB_DAT @0x60215fe8 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Warning: Global Symbol __cxa_finalize not found, cannot apply R_386_GLOB_DAT @0x60215fec ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: Global Symbol program_invocation_short_name not found, cannot apply R_386_GLOB_DAT @0x60215ff4 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Warning: Global Symbol _ITM_registerTMCloneTable not found, cannot apply R_386_GLOB_DAT @0x60215ff8 ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: Global Symbol environ not found, cannot apply R_386_GLOB_DAT @0x60215ffc ((nil)) in /lib/i386-linux-gnu/libbsd.so.0
Error: PltResolver: Symbol fstat(ver 4: fstat@GLIBC_2.33) not found, cannot apply R_386_JMP_SLOT 0x30004004 (0x30001046) in /tmp/a.out

Note that I don't use bundled x86lib's from the repository, namely libstdc++6 breaks my working version of it which I use in qemu.

[ValdikSS]

I figured out how to add wrapping symbols to libc.

@ptitSeb, do I understand correctly that right now the linker does not respect symvers for wrapped libraries, that's why there are checks for glibc version? I see the version checks in the loader itself (for emulated libraries), but not in the wrappers.

[ptitSeb]

I figured out how to add wrapping symbols to libc.

@ptitSeb, do I understand correctly that right now the linker does not respect symvers for wrapped libraries, that's why there are checks for glibc version? I see the version checks in the loader itself (for emulated libraries), but not in the wrappers.

There is some limited support for symver, but most wrapped function ignore that. You can create a GO2(blah@VER2, iFp, my_blah_old) for example to hande some symver.

[ValdikSS]

So, right now I've just uncommented these stat, lstat, fstat and made them GO

box86/src/wrapped/wrappedlibc_private.h

Lines 2268 to 2273 in 027605e

	// Those symbols don't exist in non-Android builds
	//GOM(__libc_init,
	//GOM(stat,
	//GOM(lstat,
	//GOM(fstat,
	//GO(__errno,

Not sure how to fix this properly as this apparently would break Steam userspace you're targeting. These symbols were wrapped via __xstat in previous glibc versions.

[ptitSeb]

So, right now I've just uncommented these stat, lstat, fstat and made them GO

box86/src/wrapped/wrappedlibc_private.h

Lines 2268 to 2273 in 027605e

	// Those symbols don't exist in non-Android builds
	//GOM(__libc_init,
	//GOM(stat,
	//GOM(lstat,
	//GOM(fstat,
	//GO(__errno,

Not sure how to fix this properly as this apparently would break Steam userspace you're targeting. These symbols were wrapped via __xstat in previous glibc versions.

A GOM should be created and check if stat symbol exist. If yes, use it, else use __xstat...