Help wanted: VM/Hypervisor hangs on boot when a tap network interface is configured

[nagisa]

Describe the bug

When I boot a systemd-based VM with a TAP network interface enabled, I find that somewhere along the boot the VM stops making progress. I see 100% CPU usage on the hypervisor process. The guest linux kernel sometimes prints

[   27.438419] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[   27.439045] rcu: All QSes seen, last rcu_preempt kthread activity 21002 (4294694586-4294673584), jiffies_till_next_fqs=3, root ->qsmask 0x0
[   27.440236] rcu: rcu_preempt kthread starved for 21002 jiffies! g20361 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
[   27.441257] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[   27.442136] rcu: RCU grace-period kthread stack dump:
[   27.442797] rcu: Stack dump where RCU GP kthread last ran:

If I boot this same VM with qemu or crosvm, using exactly the same guest and equivalent hypervisor settings, the guest boots successfully. This suggests to me that an issue is more likely in cloud-hypervisor, but I cannot really tell for sure. Similarly if I just remove the --net argument the guest boots with cloud-hypervisor just as well.

Unfortunately I don’t really have the knowledge necessary to figure out how to investigate this further, and hence the help request.

To Reproduce
Steps to reproduce the behaviour: Will try to come up with something if there is no way for me to investigate this by myself.

Version

Output of cloud-hypervisor --version: 33.0

Did you build from source, if so build command line (e.g. features):

VM configuration

What command line did you run (or JSON config data):

sudo /nix/store/28wd1f0qndyd9x5p410b96lxjggphjmq-cloud-hypervisor-33.0/bin/cloud-hypervisor --cpus boot=1 --watchdog --console null --serial tty --kernel /nix/store/w3shfnmgpl94jmg9byv7mdsx2gp5kacl-rust-hypervisor-firmware-x86_64-elf-0.4.2/bin/hypervisor-fw --cmdline console=ttyS0 reboot=t panic=-1 init=/nix/store/xa83ssy8qqjs3b1hgn32b7sclpqjc79p-nixos-system-unifi-netserv-23.11.20230817.42c2560/init --seccomp true --memory mergeable=on,shared=on,size=1024M --disk path=/nix/store/0j5162xrmqkjfg6zlw31kcd2wv69j1a6-microvm-bootdisk.img,readonly=on --fs tag=unifi-netserv-var-lib-unifi-data,socket=unifi-netserv-virtiofs-unifi-netserv-var-lib-unifi-data.sock --fs tag=journal,socket=journal.sock --fs tag=ro-store,socket=unifi-netserv-virtiofs-ro-store.sock --api-socket unifi-netserv.sock --net tap=unifi-netserv0,mac=00:02:00:01:01:01

On the host side I created the TAP interface as such:

sudo ip tuntap add unifi-netserv0 mode tap user root

I also tried --net tap=,mac=,ip=,mask= as presented in the README as well as iommu=on/off but the result was the same in either case.

Guest OS version details: An "empty" NixOS 23.11 system built from sources from 2 weeks ago or so
Host OS version details: NixOS 23.11 system built from sources from 2 weeks ago

In either case linux kernel is 6.1.46.

Logs

Output of cloud-hypervisor -v from either standard error or via --log-file: log.txt

Linux kernel output: no logs from the host kernel as far as I can tell.

[rbradford]

In the log output (including the serial) it looks like it boots fine - I see that you're using --console null and --serial tty but specifying console=ttyS0 on the command line - is it possible your getty is running on the virtio-console which is why it's not appearing for you. Use --console off to disable that.

[nagisa]

is it possible your getty is running on the virtio-console which is why it's not appearing for you

The thing is – my getty shows up perfectly fine on the serial interface (and hypervisor’s CPU utilization stays at a reasonable level of 0%) if the --net option is removed. Presence or absence of --net option shouldn't affect how getty works, should it?

Regardless changing --console null to --console off did not change anything, unfortunately.

I have also looked at the guest’s journal (by putting it on a shared filesystem), but it does not really have anything suspicious either.

All things considered, if my deduction that this is a hang somewhere, is correct, then I wouldn't expect systemd or similar logs to show anything terribly wrong either.

[Geo-7]

I have the exact same problem
host OS = Alpine Linux edge kernel 6.6.4-1-lts
cloud-hypervisor v36.0.0

cloud-hypervisor \
        --kernel ./hypervisor-fw \
        --disk path=jammy-server-cloudimg-amd64.raw \
        --cpus boot=4 \
        --memory size=2048M \
        --net "tap=" \
        --seccomp false \
        --log-file net.log \
        -v

It boots fine without --net

[thomasbarrett]

I have also experienced this issue when using a virtio-net devices. The cloud-hypervisor integration tests worked on my machine, but using a virtio-net with a stock ubuntu 22.04 image did not. This indicates that the issue is with triggered by specific guest behavior. Perhaps there is a virtio-net feature flag being negotiated that isn't fully supported and/or tested be cloud-hypervisor.

In my case, the guest kernel logs indicated a hang in the virtio-net client code. I don't use virtio-net (and am not entirely familiar with the spec), so I haven't looked into this too deeply.

[rbradford]

Yes, this is a known issue in some kernels used on Ubuntu 22.04 - #5998

[thomasbarrett]

👀 ty!