Environment variables are not properly shared between processes

[kosticmarin]

Issue was first noticed from a failing test config::config_env_variable

First I've made sure to check that the issue wasn't introduced by wasmtime version changes, the issue is still present
when using versions 3, 4 and 5.

I was unable to find the cause and fix this issue, but I'll present my findings below.

Expected behavior

Lunatic runtime will always create a single process when executing a wasm, this process will inherit all environment
variables from the host machine. Users are allowed to create other processes and control if these processes
will inherit environment variables or provide each process with a custom ProcessConfig which will hold custom
environment variables.

e.g. using the Lunatic Rust Library

use lunatic::{spawn_link, Mailbox, ProcessConfig};

fn print_env_vars() {
    for (k, v) in std::env::vars() {
        println!("{k}={v}");
    }
}

#[lunatic::main]
fn main(_: Mailbox<()>) {
    println!("PARENT");
    print_env_vars();

    let task = spawn_link!(@task || {
      println!("CHILD INHERITS");
      print_env_vars();
    });
    let _ = task.result();

    let mut config = ProcessConfig::new().unwrap();
    config.add_environment_variable("hello", "world");
    config.add_environment_variable("foo", "bar");

    let task = spawn_link!(@task &config, || {
      println!("CHILD CUSTOM");
      print_env_vars();
    });
    let _ = task.result();
}

From the example above we expect that the parent process and the first child process
will have the same environment variables, while the second child process will not.

Under the hood

So lets see what exactly happens when the code above is executed and how does Lunatic runtime
manage environment variables.

Each process has its own state (DefaultProcessState), this state holds the WasiCtx.
This context is the "state" to the WASI host function interface and it stores environment variables.

lunatic/src/state.rs

Line 64 in 07454fb

wasi: WasiCtx,

WasiCtx and its configuration is either copied from the parent process or is defined manually by creating the ProcessConfig.

lunatic/crates/lunatic-process-api/src/lib.rs

Line 559 in 07454fb

let config = match config_id {

lunatic/crates/lunatic-process-api/src/lib.rs

Line 581 in 07454fb

let mut state = state.new_state(module.clone(), config)?;

lunatic/src/state.rs

Line 141 in 07454fb

wasi: build_wasi(

Finally when the wasm calls a WASI host function, e.g. std::env::vars(), the Linker makes sure that the correct wasi context is used.

lunatic/crates/lunatic-wasi-api/src/lib.rs

Line 52 in 07454fb

|ctx| ctx.wasi_mut(),

I've check and traced the Lunatic execution and can confirm that passing around the config and environment variables is correct.
That the correct WasiCtx is called from each process.

But in the end the host call fails returning empty array. What is even more strange there is a difference how lunatic::mode::cargo_test and lunatic::mode::execution behave.

In normal execution std::env::vars will correctly return values only for the first process spawned, in all other processes spawned from the first process the std::env::vars call will return an empty array.

In test execution mode std::env::vars always returns an empty array.

Did the unit test ever work?

Also I ran the test suite against the different versions of the runtime.
I'm running this on Linux 5.15.0-58-generic Ubuntu 22.04 with, Rust Version 1.67.0.

Lunatic Runtime	Lunatic	Test
main latest commit	main latest commit	❌
v0.10.1	v0.10.6 (7ee456cb2c3f8461a7fb6d24877f0eb29e991233)	❌
v0.10.0	v0.10.6 (7ee456cb2c3f8461a7fb6d24877f0eb29e991233)	❌
v0.9.2	v0.9.0 (83943192195c10b4e7eac01ab09c91eff0c242bd)	❌
v0.9.0	v0.9.0 (83943192195c10b4e7eac01ab09c91eff0c242bd)	❌
v0.9.0	v0.8.0 (10d1a9e3dafc92499aa9c3911c911e1b3a093076) first added	❌

Does the wasmtime even work

I've written the smallest example of embedded wasmtime runtime and it seems to work.

Source is available here.

Conclusion

What I managed to find out is that the system calls don't even happen. By modifying the original source from wasmtime (wiggle generate_func) I was able to produce this output for the example above. We can see that the parent processes calls three functions in order to execute std::env::vars() while the child process just returns empty array without actual syscalls. But still I can't seem to figure out what is the cause of this behavior.

[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Spawning process: 1
lunatic::runtimes::wasmtime::instantiate
wasmtime::instance::pre_instantiate_raw
[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Process size: 1096
[2023-02-02T13:52:47Z TRACE lunatic_process] Process 1 spawned
lunatic::runtimes::wasmtime::call
wasmtime::func::call_impl
wiggle::generate::linker::func_wrap environ_sizes_get
wiggle::generate::linker::func_wrap environ_get
wasi-common::snapshots::environ_get
wiggle::generate::linker::func_wrap fd_write
PARENT
wiggle::generate::linker::func_wrap fd_write
ALACRITTY_LOG=/tmp/Alacritty-4620.log
...
...
[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Spawning process: 2
lunatic::runtimes::wasmtime::instantiate
wasmtime::instance::pre_instantiate_raw
[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Process size: 1096
[2023-02-02T13:52:47Z TRACE lunatic_process] Process 2 spawned
lunatic::runtimes::wasmtime::call
wasmtime::func::call_impl
wiggle::generate::linker::func_wrap fd_write
CHILD INHERIT
[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Spawning process: 3
lunatic::runtimes::wasmtime::instantiate
wasmtime::instance::pre_instantiate_raw
[2023-02-02T13:52:47Z TRACE lunatic_process::wasm] Process size: 1096
[2023-02-02T13:52:47Z TRACE lunatic_process] Process 3 spawned
lunatic::runtimes::wasmtime::call
wasmtime::func::call_impl
wiggle::generate::linker::func_wrap fd_write
CHILD CUSTOM

[bkolobara]

Just for reference. This is caused by rust-lang/rust#107635