Youki does not have reply from dbus

[peter-jerry-ye]

Hi, I'm trying to use youki to run wasm. I followed the tutorial, using the rootful podman + debug build of youki + feature wasmtime. However, I'm getting the following error:

DEBUG youki: started by user 0 with ArgsOs { inner: ["/home/zihang/Document/youki/youki", "--log-level=trace", "delete", "--force", "ab1a269dbf8d7b0d7f74c8738b3b72dfe35328675d9b792f9c6b425ddc8f4740"] }
DEBUG youki::commands::delete: start deleting ab1a269dbf8d7b0d7f74c8738b3b72dfe35328675d9b792f9c6b425ddc8f4740
Error: OCI runtime error: /home/zihang/Document/youki/youki: thread 'main' panicked at crates/libcontainer/src/process/container_intermediate_process.rs:49:82:
called `Result::unwrap()` on an `Err` value: Systemd(SystemdClient(DBus(MethodCallErr("expected to get a reply for method call, didn't get any"))))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: failed to receive. "waiting for intermediate process". BrokenChannel

Caused by:
    channel connection broken

What may be the cause of the error? Thank you.

[utam0k]

Thanks for your report. May I ask you to build youki from the latest source code? I guess we have already fixed this bug 👍

[peter-jerry-ye]

Thank you for your reply. I built it from the main branch, and I was running rootful podman, for which I didn't find any similar issues.

[peter-jerry-ye]

Also, I can't reproduce it every time. Sometimes, it gives an error message like this:

DEBUG youki: started by user 0 with ArgsOs { inner: ["/home/zihang/Document/youki/youki", "--log-level=trace", "delete", "--force", "ffbf349d69bede6871c58efbdd4e53fe37f0021dec0b63c9409c54e7e76d9824"] }
DEBUG youki::commands::delete: start deleting ffbf349d69bede6871c58efbdd4e53fe37f0021dec0b63c9409c54e7e76d9824
Error: OCI runtime error: /home/zihang/Document/youki/youki: Error: failed to receive. "waiting for init ready". BrokenChannel

Caused by:
    channel connection broken

[YJDoc2]

Hey @peter-jerry-ye , are you able to run podman rootless container with youki , or not at all? The channel connection broken error is probably a side effect of dbus error. I have seen this in CI a few times as well, but not able to reproduce it locally.

[peter-jerry-ye]

@YJDoc2 Hi. Yes, I can run podman normally with youki unless it's a wasm workload. I think it is the wasm workload that causes the problem, but I couldn't find any addition error messages to help me locate the problem.

[YJDoc2]

@peter-jerry-ye while it is possible that it is specifically wasm workload that is causing the issues ; I feel this is a more general issue, as in our CI (where I have also seen this kind of failure) does not test wasm workload.

Can I ask you to run non-wasm workloads in rootless mode in a loop for sometime, and see if that also fails? And would you be willing to share strace and dbus logs related to youki when such failure happens, so we might be able to understand this better? If you're fine with sharing them, I'll comment on commands to be run, and what exactly is to be collected. Thank you!

[peter-jerry-ye]

I just retried another way. I built with sudo buildah build --annotation "module.wasm.image/variant=compat" --platform wasi/wasm -t wasm-module . and it worked with sudo podman --runtime ~/Document/youki/youki run --rm localhost/wasm-module it worked. The question is, did it run with youki or crun-wasm?

[peter-jerry-ye]

@YJDoc2 No, I did not observe failure with non-wasm workloads. Could you please let me know the commands to be run? Thanks.

[YJDoc2]

Hey, so not sure why this would specifically happen for wasm workloads, but can you run the following commands and share the o/p?

# podman create command with --runtime youki

strace -o ./strace.log -e '!futex' -x  -v -s 2048 -f podman run <container_name>

The create command should be run normally, as podman does not call youki there ; running the strace + podman run should create a strace.log file, which should contain the dbus messages sent by youki. Note if you get error like cannot open strace.log permission denied , run this in /tmp dir instead.

Apart from that another useful log would be of dbus messages itself, which can be seen by running dbus-monitor --address unix:path=/run/user/id/bus (replace id with your uid) . The issue is many application use this bus, so you get a lot of noise here. For example, simply switching from one application to other using alt+tab can log several messages.

The way I do this is to create two terminals, scale them so both are visible at same time and I don't need to switch windows ; try to keep any other programs running to minimum ; let the initial message flood for few seconds, and then running the podman command. That way the next messages should all be of youki itself. If you can share those logs as well, it wold be great!

Thanks a lot!

[peter-jerry-ye]

strace.log

Hi, here's my strace.log. Though I think it reported a different error this time.

[YJDoc2]

@peter-jerry-ye For some reason, I cannot access the log file you have uploaded. It shows a 404 error when I open the link. Can you check / re-upload? Also can you tell what different error it was ? That would also help when going through the strace logs. Thank you!

[YJDoc2]

Hey @peter-jerry-ye , can you take a look at my previous comment? Thanks!

[peter-jerry-ye]

@YJDoc2 Sorry, I had some health issue during the last few weeks. I'll reupload the strace.
strace.log