Perfomance of XCUITests interacting with System dialogs on Sonoma VM

[matzuk]

Hello everyone!

I've noticed that engineers have struggled with VM performance on Sonoma, especially with its initial versions. The performance has improved over time, but I'm still observing some drastic performance pitfalls, particularly when XCUITests interact with system dialogs, such as the Permission dialog for Location, etc. These interactions can cause tests to hang for minutes at a time (1, 2, 5, 10, or more).

I suspect the issue arises because XCUITest fails to detect the system's Idle state. Has anyone else experienced similar problems? How have you managed to overcome them? What are your thoughts?

[nathreed]

I have observed similar problems with XCUITests on both Sonoma and Ventura runners. Generally, I observe extraordinarily high CPU usage, far more so than when the tests run not in a VM on the same machine. Not sure if that’s the same issue or what’s causing your tests to hang for so long.

My VMs run on a Mac Studio with 20 cores and 128GB of RAM, and since you can only have two at a time, my VMs have 9 CPUs and 63GB RAM each, so it absolutely should not be a resource issue.

I have tried many things including using https://github.com/biscuitehh/yeetd to address the high CPU usage without success. I observe similar symptoms with tests hanging for a long time and if they do run, being very flaky.