Initial AudioVM implementation with pipewire and pulseaudio

[josa41]

Initial version of AudioVM with Pipewire backend and pulseaudio TCP remote communication layer for the guest VMs. Note that this is not really secure design (yet) basically all VMs can access the pulseaudio TCP service.

Description of changes

Separate AudioVM with HW passthrough audio device

• Pipewire backend with pulseaudio tunnel for VM client

Checklist for things done

• [X ] Summary of the proposed changes in the PR description
• [X ] More detailed description in the commit message(s)
• [X ] Commits are squashed into relevant entities - avoid a lot of minimal dev time commits in the PR
• [X ] Contribution guidelines followed
• Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
• PR linked to architecture documentation and requirement(s) (ticket id)
• [X ] Test procedure described (or includes tests). Select one or more:
  • [X ] Tested on Lenovo X1 x86_64
  • Tested on Jetson Orin NX or AGX aarch64
  • Tested on Polarfire riscv64
• [X ] Author has run nix flake check --accept-flake-config and it passes
• All automatic Github Action checks pass - see actions
• Author has added reviewers and removed PR draft status

Testing

• 3.5mm headset works
• Integrated speaker works, automatically swapping to/from 3.5mm IO
• Tested with Chromium, Element should also work
  Notes:
• Integrated speaker volume is a bit low
• Integrated mic does not work (possible kernel/driver issue)
• USB/BT sound devices does not work (will be done separately)

[leivos-unikie]

Test results on Lenovo-X1-gen11

manual testing

• Poweroff and reboot buttons don't work
• All apps launch by clicking their icons
• Audio output works both from integrated speakers and 3.5mm headphones, automatically swapping to/from 3.5mm IO
• Audio output works both in chromium and element
• Integrated microphone does not work
• Youtube video real playback time does not differ from the video nominal lenght
• Increasing volume works in audio-vm with “pactl set-sink-volume @DEFAULT_SINK@ 48000”

ci-test-automation

• performance ok
• bat tests passed

[josa41]

Reboot & shutdown issue fixed. The issue was that Polkit package was removed from host along with pulseaudio. Apparently only pulseaudio had a dependency for Polkit so polkit was removed from build. Manually enabling polkit on host fixed the issue.
Will add separate tickets for adding USB sound device passthroughs and fixing integrated mic issue.
Also there is a separate ticket to add audio controller app in guivm which fixed the low volume issue.

[leivos-unikie]

Now reboot and poweroff buttons work.

[leivos-unikie]

Tested now also audio input with 3.5mm mobile phone headphones. Doesn't work. (Verified that the mic of the headphones work with my nixos work laptop.

[leivos-unikie]

Tested 3.5mm mobile phone headphones mic again with instructions from Jon.

• It works if plugged before chromium window is opened.
• If headphones is plugged in when chromium window is already open the mic doesn't work. In this case closing all chromium windows and then opening a new chromium window sets mic working.

[leivos-unikie]

Tested that 3.5mm headphones mic works also in Element app Jitsi Video Conference but only if plugged before pressing the button "Join Conference".

[leivos-unikie]

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active. Laptop power led did shut down but two leds stayed on:

• speakers off / F1
• mute / F4

Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening. Only after doing hard reset (pressing the power button 15 sec) I was able to boot the laptop.

Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

[milva-unikie]

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active. Laptop power led did shut down but two leds stayed on:

* speakers off / F1

* mute / F4

Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening. Only after doing hard reset (pressing the power button 15 sec) I was able to boot the laptop.

Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

[vilvo]

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active.
...
Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening.
...
Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

#565 could be adapted to log from relevant VMs to persistent storage to capture what's going on here. See Testing-section in the #565 PR description.

[leivos-unikie]

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active.
...
Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening.
...
Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

#565 could be adapted to log from relevant VMs to persistent storage to capture what's going on here. See Testing-section in the #565 PR description.

I tried rebasing on that log-vm and solving few merge conflicts. But the build fails.

[leivos-unikie]

Tried to see journalctl -b-1 on ghaf-host after shutdown with 3.5mm mic on in chromium-vm
No journal boot entry found from the specified boot offset (-1).

After normal shutdown journal -b-1 shows logs from the previous boot.

Then tried to leave this running on ghaf-host while doing the shutdown test:
journalctl -f > /home/ghaf/host-journalctl.log &

but there was nothing special, just repeating these:

Apr 30 13:00:55 ghaf-host vsockproxy[898]: Connection 6 RX 7132 bytes in 1.00 seconds, transfer rate 0.01 MB/s, rx total 185952
Apr 30 13:00:55 ghaf-host vsockproxy[898]: Connection 6 TX 29677456 bytes in 1.00 seconds, transfer rate 29.68 MB/s, tx total 353325956
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 5 RX 21391864 bytes in 1.00 seconds, transfer rate 21.38 MB/s, rx total 374717820
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 5 TX 4556 bytes in 1.00 seconds, transfer rate 0.00 MB/s, tx total 190508
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 6 RX 4556 bytes in 1.00 seconds, transfer rate 0.00 MB/s, rx total 190508
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 6 TX 21391864 bytes in 1.00 seconds, transfer rate 21.38 MB/s, tx total 374717820

[leivos-unikie]

Tested 'shutdown with 3.5mm headset mic active' issue also with current ghaf mainline build (himalia 82). This issue is not in the mainline.

[josa41]

The shutdown issue is a great finding. I can reproduce it also by starting the microphone/recording in chromium and then going to ghaf-host and using systemctl to stop audiovm. You can see that the recording in chromium stops (i use mic-test.com). Now audio-vm is turned off and shutting down the laptop seems to cause the issue, some times the indicator lights turn off. Checking that the fan at the bottom keeps spinning and also the laptop does not turn on with a short press of power button (needs the 15second shutdown first).
It seem the issue is between audioVM and chromiumVM connecting getting hanged by audio not responding.
It could also be audio hardware (or one of the audio related passthrough devices) getting stuck for audio VM shuttin down.
Anyway shutting down just audioVM may allow us easier way of getting some logs of what is going on.

[josa41]

I was able to reproduce this with just audiovm so ssh to audiovm and record audio with pulseaudio tools
parecord -r output.wav

Now stop microvm@audio-vm.service from host and click the shutdown icon from guivm.
The audio indicator lights in most cases stay on and some cases the X1 bottom fan keeps spinning even when the lights may turn off.

[josa41]

Figured out a way to reset the audio device on host after audio-vm has shut down and audio device is got in some buggy state. This can be demoed as below:

So after ssh to audio-vm and starting a recording

parecord -r output.wav

ssh to host and use systemctl to shutdown audio-vm to get to error state
After the audio-vm shutdown (when audiovm is recording) turning off the laptop will result in the bottom fan and parts of the laptop staying on..

This can be fixed from host by forcing a reset on the pci device (remove device and rescan bus):

sudo su
echo "1" /sys/bus/pci/devices/0000:00:00:1f.3/remove
sleep 1
echo "1" /sys/bus/pci/devices/0000:00:1f.0/rescan

Next step is figure out how to automate this..

[Mic92]

So by just looking at the code, something like the following should allow to run code after the service has started:

systemd.services."microvm@audio-vm".serviceConfig = {
  # The + here is a systemd feature. I tmake the script run as root.
  ExecStartPost = ["+${pkgs.writeShellScript "reload-audio" ''
    echo "1" > /sys/bus/pci/devices/0000:00:00:1f.3/remove
    sleep 1
    echo "1" > /sys/bus/pci/devices/0000:00:1f.0/rescan
  ''}"];
};

Now the sleep command might be a bit racy. Maybe you can find some condition that you can check if the 1s was not enough and you have to retry. Another issue with the postStart is figuring out the readiness of the hypervisor. It looks like cloud-hypervisor has support for systemd-notify socket notification but I think this one is using qemu? Qemu has also some control sockets it exposes like qmp, that could be used to check for readiness.

Hope that helps.

[Mic92]

Ok. Here is an simple idea: Use socat in the VM over VSOCK to communicate with a socat outside the VM, when the recording starts. Than you can run the fix-up command just in the right time by blocking with socat. Systemd now also add service-level synchronization for their systemd-nspawn integration, but no idea in what state that is.

[Mic92]

Ok. So for context, @josa41 found out that the resets brings back the audio device power state from D3 into D0.

[josa41]

Thanks @Mic92, I got the fix done and it seems to work. Audio device power state stays at D0 and with the fix the laptop shuts down properly. I did lower the sleep to 0.1 so it is not noticeable by user. Lets hope it does not cause issues with VM restart (don't tell our testers) ;)

[josa41]

Tested sound and audio recording with chromium

• Microphone works only with 3.5mm headset
• Laptop shutdown issue fixed

[leivos-unikie]

Tested first the laptop shutdown issue with 3.5mm headset mic plugged

• with chromium, ok
• element app does not detect any microphone (Before the fix I was able to activate mic in Jitsi Video Conference in Element app.)

[leivos-unikie]

Test results on Lenovo-X1

• ci-test-automation bat tests passed / performance ok
• All apps launch
• No shutdown issues
• Audio input/output works in chromium with 3.5.mm headset
• Audio input/output does not work in element app, not with 3.5mm headset nor with integrated speakers/mic

[josa41]

It seems element-vm network setup is done so that it does not resolve audio-vm.ghaf or other ghaf internal dns addresses. It does have access to ghaf addresses with direct ip address.
See dns settings in element-vm.ghaf:
systemd-resolve --status

[leivos-unikie]

Rebased josa41 audiovm branch on josa41 origin/element-dns-fix with
git rebase origin/element-dns-fix

to be able run tests for both PRs #576 and #600 merged together.

Test reults on Lenovo-X1:

• Now 3.5mm headset audio input/output works in Element App
• Intergrated speakers work.
• Integrated mic does not work. Logi USB headset does not work. (this was expected, will be fixed separately)
• All apps launch from icons
• ci-test-automation tests pass and performance test are ok