Harden fuzzing with respect to non-deterministic behaviors between Wasm engines

[Robbepop]

One big problem of our current differential fuzzing is that it reports false positives, i.e. it reports bugs for cases where the 3 Wasm engines, Wasmi (register), Wasmi (stack) and Wasmtime behave differently but all of them correctly.

One example is a function that requires a lot of stack space which oftentimes leads to a stack overflow Wasm trap for Wasmi (register), sometimes for Wasmi (stack) and only very rarely for Wasmtime. The reason for this is two fold:

• Wasmi (register) uses fixed sized memory sizes per function call whereas Wasmi (stack) has a ... stack that grows dynamically. Therefore Wasmi (register) usually reaches the point of stack overflow a bit faster.
• Wasmi (register) and Wasmi (stack) both do not optimize the input Wasm nearly as good as Wasmtime which makes it possible for Wasmtime to shrink the required amount of stack space enough to usually require less stack space during execution of maliciously bad test cases.

Locally I ran both Wasmer (Cranelift) and Wasmer (Singlepass). One would argue that both should behaves very similarly. However, as described above Wasmer (Cranelift) applies many more optimizations than Wasmer (Singlepass) which allowed me to observe similar behavioral differences as with Wasmi (register) vs Wasmtime with respect to stack overflow non-determinism.

Ideally we want our fuzzing to resist these false positives that are connected to non-deterministic memory profiles of different Wasm engines.