M1 Laptops cannot run all features of govuk-docker

[KludgeKML]

An issue to track findings on running govuk-docker on M1 Laptops.

Current issues:

• When initially running a 'make ' the make fails when trying to install google-chrome-stable. This can be replaced with chromium to get past the initial step, but tests runners later may rely on google-chrome-stable and cause failures in test suites.
• MySQL versions below 8 are not available in arm64 architectures (possibly replace with MariaDB?)
• MongoDB versions below 3.6 are not available in arm64 architectures (so router and router-api cannot be run. 3.6 is probably a safe replacement, since we're going to try to upgrade the production database to that anyway)

[ollietreend]

MySQL versions below 8 are not available in arm64 architectures

The good news here is that none of the GOV.UK apps are dependent on MySQL < 8.

There's a ticket on the database migration board to remove the MySQL 5.6 service from the docker compose configuration. Right now it's still defined but unused/redundant.

[kevindew]

Thanks for opening the issue here and sorry that this issue has caught you at the time of you onboarding. I've put in a request with IT for an M1 machine myself to see if I can help (or at least experience the pain)

[Tetrino]

Additional:

• inotify does not run under QEMU and will thus struggle in docker engine when running in AMD64 compatibility mode. I am unaware if there is an ARM build for it however if there is, our docker installation does not grab the relevant gems for the platform. It appears that Docker might use QEMU to run ARM on M1 chips anyway currently, so we may need to find a replacement.
• govuk_test gem fails to run correctly due to being unable to call chrome driver via the webdrivers gem. We might be able to get ahead of this by updating the webdrivers gem value.
• rabbitmq is best run on ARM64 mode rather than through the AMD64 QEMU due to known instability when emulated.
• Replication scripts do not succeed without already having built and run the images we're cloning to first despite the behaviour being to tear down all the running containers. The images need to be initialised to, for example, setup the postgresql database in publishing-api. This is a regression from current behaviour.
• Even if we manage to get everything up and running, any *.dev.gov.uk endpoints do not route correctly and cannot find the containers. Curls are intermittent, which makes me think it's a pipeline issue?

Seems a lot of this we can fix by upgrading to the latest version of the gems we rely on, however some of it may be bespoke for our environment.

[KludgeKML]

Per @Tetrino 's suggestion I've made a branch that we can use as a fake main for M1 changes:

https://github.com/alphagov/govuk-docker/tree/m1-investigation

If we make M1-related changes, we can branch from there and PR into there and we'll have a handy way of 1) getting the fixes we need and 2) comparing the changes with main.

[KludgeKML]

#564 - PR for fixing the dnsmasq problem.

[KludgeKML]

Note for related apps

Apart from the things required to get govuk-docker working at all, any app you want to run needs to have:

• ruby bumped to min 2.7.5
• the gems spring and spring-watcher-listen commented out in the gemfile if present.
• the listen gem disabled by commenting out the line config.file_watcher = ActiveSupport::EventedFileUpdateChecker in /config/environments/development.rb

(The last two items remove spring and inotify autorestart in development mode, so they're not entirely ideal. Possibly there should be some boilerplate to allow them on intel systems).

[KludgeKML]

govuk_test gem fails to run correctly due to being unable to call chrome driver via the webdrivers gem. We might be able to get ahead of this by updating the webdrivers gem value.

This looks like it's linked to inotify. If you bash into a running app's container and try to run google-chrome-stable, it fails with a coredump about inotify:

root@35558c032be5:/govuk/static# /usr/bin/google-chrome-stable --no-sandbox
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
[370:370:0131/164025.878444:ERROR:nacl_fork_delegate_linux.cc(329)] Bad NaCl helper startup ack (0 bytes)
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
[334:379:0131/164025.994483:ERROR:file_path_watcher_linux.cc(321)] inotify_init() failed: Function not implemented (38)
[0131/164026.063886:ERROR:scoped_ptrace_attach.cc(27)] ptrace: Function not implemented (38)
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped

[kevindew]

Are these problems when trying to run the tool through a compatibility tool (rosetta or similar) and an x86 architecture or are you compiling the images with an arm64 arch?

[KludgeKML]

These are when I force amd64 builds for the app container (not necessarily on the other containers) by

# Install packages for building ruby
FROM --platform=linux/amd64 buildpack-deps

(So they're running through rosetta, I believe)

Without the --platform, google-chrome-stable won't install at all, with it it installs, but then won't run (because of inotify I think). The other option is remove the --platform tag and include chromium rather than google-chrome-stable, but the webdrivers used in gov_uk test seem not to be able to find nor use chromium.

[DilwoarH]

I found that I am able to build using the following changes: #569

There's still errors for me to work through but build seems to "mostly" finish

[KludgeKML]

@DilwoarH I think you can simplify that slightly by just editing Dockerfile.govuk-base and changing the line

# Install packages for building ruby
FROM buildpack-deps

...to....

# Install packages for building ruby
FROM --platform=linux/amd64 buildpack-deps

# Set M1 Workaround variables
ENV DISABLE_SPRING true

This forces all the app images to be built in amd64 (rather than specifying each one in its compose file), and turns off spring in any app that uses it (because spring relies on inotify, which doesn't work under qemu).

For any individual app you want to run you'll then need to bump it to ruby 2.7.5 (uncontroversial, because lots of them are being bumped to that anyway at the moment), and disable the listen gem (which uses inotify). I haven't come up with a nice way to disable listen at the base the way spring is disabled, sadly.

[Tetrino]

Another option is to use amd64/buildpack-deps instead of calling FROM at that point. We can also force docker to run in AMD64 mode as default platform using DOCKER_DEFAULT_PLATFORM=linux/amd64 however I agree the dockerfile is the better location.

[DilwoarH]

PR to fix the chrome issues: #576

[kevindew]

I found I didn't need to set anything up to build things for amd64 - it just worked like that by default?

[KludgeKML]

Current state of affairs (mostly good):

Thanks to everyone's hard work, we're now at a stage where mostly everything works on M1 laptops. If you have one, you can proceed just as everyone else does, with the following exceptions:

• Until govuk_tests is updated, you will need to tell it not to try to find drivers by adding the following lines to the bottom of your /Dockerfile.govuk-base (and the Dockerfile in the /projects/whitehall and /projects/asset_manager project folders)

  # Tell the govuk_test gem not to try to load webdrivers
  ENV GOVUK_TEST_USE_SYSTEM_CHROMEDRIVER yes

• The default docker MySQL-8 app does not have an ARM64/v8 image. You can use an official MySQL image by replacing the mysql-8 image with mysql/mysql-server:8.0.28-aarch64 in /docker-compose.yml

• The default docker ES-6 app does not have an ARM64/v8 image. You can replace it with ES-7 with no ill effects (that I've spotted yet) by replacing elasticsearch-6 dependencies in (eg) /projects/licence_finder/docker-compose.yml (affects: licence-finder, search-api)

• Mongo 2.6 is not supported. You can replace it with Mongo 3.6 with no ill effects by replacing mongo2.6 with mongo3.6 in (eg) /projects/router/docker-compose.yml (affects: router, router_api)

• govuk-puppet can't run, because it requires Ruby 1.9.3 at the moment (not supported)

Not govuk-docker, but to record somewhere:

• if you want to use terraform, you can't use the default brew terraform. You need to install the hashicorp tap:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

[ollietreend]

@KludgeKML

Until govuk_tests is updated, you will need to tell it not to try to find drivers

We released govuk_test v3.0.0 today 🚀

The GOVUK_TEST_USE_SYSTEM_CHROMEDRIVER workaround won't be needed once apps have been updated. And despite it being a major release, dependabot should pick it up because our repos don't specify a version constraint.

[kevindew]

I've put together a branch that should contain all the known fixes (thanks @KludgeKML, they're mostly yours) that collates what we know so far: #581