Test failures on Mac in CI

[joeyparrish]

Tests are failing or failing to run on Mac, but only in CI. Everything works in the lab.

The last build-and-test job to succeed was this one for the 4.8.0 release PR:
https://github.com/shaka-project/shaka-player/actions/runs/8819839697/job/24212073498

Since then, it always fails, with errors like this for Firefox and Edge:

29 04 2024 21:12:10.311:ERROR [launcher]: Cannot start Firefox via WebDriver
	Can not find the binary /Users/runner/.webdriver-installer-cache/geckodriver
	Please set env variable GECKODRIVER_PATH
29 04 2024 21:12:10.[31](https://github.com/shaka-project/shaka-player/actions/runs/8884399216/job/24396860329?pr=6505#step:9:32)2:ERROR [launcher]: Firefox via WebDriver stdout: 
29 04 2024 21:12:10.312:ERROR [launcher]: Firefox via WebDriver stderr: 
29 04 2024 21:12:10.312:ERROR [Firefox via WebDriver]: Failed to find driver for Firefox
29 04 2024 21:12:10.312:ERROR [Firefox via WebDriver]: /Users/runner/.webdriver-installer-cache contains: [
  "chromedriver"
]

29 04 2024 21:12:06.148:ERROR [launcher]: Cannot start MSEdge via WebDriver
	Can not find the binary /Users/runner/.webdriver-installer-cache/msedgedriver
	Please set env variable MSEDGEDRIVER_PATH
29 04 2024 21:12:06.149:ERROR [launcher]: MSEdge via WebDriver stdout: 
29 04 2024 21:12:06.149:ERROR [launcher]: MSEdge via WebDriver stderr: 
29 04 2024 21:12:06.150:ERROR [MSEdge via WebDriver]: Failed to find driver for MSEdge
29 04 2024 21:12:06.150:ERROR [MSEdge via WebDriver]: /Users/runner/.webdriver-installer-cache contains: [
  "chromedriver"
]

So webdriver-installer is unable to find installed versions of Firefox and Edge, which likely means they have moved in the CI environment.

Chrome on Mac fails tests instead, with a long cascade of timeouts like:

      ✗ sends reasonably-sized updates [Chrome 124.0.0.0 (Mac OS 10.15.7)]
	Error: Timeout - Async function did not complete within 120000ms (set by jasmine.DEFAULT_TIMEOUT_INTERVAL)
	    at <Jasmine>

These are very hard to explain. These tests passed in 4.8.0, but have failed since. Chrome on Mac has not been upgraded, as we can see the same browser & OS version numbers in the logs of the last successful run:

Chrome 124.0.0.0 (Mac OS 10.15.7): Executed 2623 of 2643 (skipped 20) SUCCESS (9 mins 18.234 secs / 9 mins 11.688 secs)

If a solution can't be found quickly, we may need to disable these in CI and rely on lab test runs to gate releases. This may result in Mac-specific regressions that are not found until release time.

[joeyparrish]

Logs of the last successful runs of Firefox on Mac and Edge on Mac:

Firefox 125.0 (Mac OS 10.15): Executed 2572 of 2643 (skipped 71) SUCCESS (8 mins 18.375 secs / 8 mins 10.21 secs)

Edge 124.0.0.0 (Mac OS 10.15.7): Executed 2623 of 2643 (skipped 20) SUCCESS (9 mins 36.587 secs / 9 mins 28.626 secs)

[joeyparrish]

Neither Edge nor Firefox appear in /Applications in CI now. 😭

Google Chrome for Testing.app
Google Chrome.app
PowerShell.app
Python 3.11
Python 3.12
Safari.app
Utilities
Xcode.app
Xcode_14.3.1.app
Xcode_14.3.app
Xcode_15.0.1.app
Xcode_15.0.app
Xcode_15.1.0.app
Xcode_15.1.app
Xcode_15.2.0.app
Xcode_15.2.app
Xcode_15.3.0.app
Xcode_15.3.app
Xcode_15.4.0.app
Xcode_15.4.app

[joeyparrish]

This could be explained by this GitHub blog post:

The macOS 12 runner image will remain latest until migration of the latest YAML workflow label to macOS 14 in Q2 FY24 (April – June 2024). While macOS 13 is now generally available under the macos-13 label, this image will not be migrated to latest. Following this announcement, macOS 11 runner image will begin deprecation immediately with retirement expected to complete by June 2024.

It is now Q2 FY24, so we may have just been auto-upgraded from macOS 12 to macOS 14 on M1, and they may not pre-install the same browsers on M1 images.

[joeyparrish]

Indeed, this table now shows:

Image	YAML Label
macOS 14	macos-latest-large or macos-14-large
macOS 14 Arm64	macos-latest, macos-14, macos-latest-xlarge or macos-14-xlarge
macOS 13	macos-13 or macos-13-large
macOS 13 Arm64	macos-13-xlarge

So if we want x86-64, we need to use macos-13 explicitly. This may not matter much for Shaka Player, but some of the native C++ projects under shaka-project may need to explicitly choose macos-13 (x64) or macos-latest (arm64) in their build matrices.

For Shaka, we will just have to explicitly install browsers now.

[joeyparrish]

Installing Firefox & Edge fixes Firefox and shows Edge now failing the same as Chrome, with a series of timeouts.

[joeyparrish]

Installing Edge & Firefox explicitly solves the failure to start tests.

Switching to macOS 13 (x64) seems to resolve the timeouts on Chrome and Edge. It's not a great permanent solution, but I think it's best for now until we understand these timeouts better.