Add basic support for Apple Silicon

[bradenmacdonald]

This PR changes the Makefile to automatically detect if the developer is using an ARM system like Apple's M1 notebooks. It will then adjust the docker-compose file so that devstack can run.

Without this PR, attempting to run devstack on an ARM mac will display the error no matching manifest for linux/arm64/v8 in the manifest list entries

Does it work?

Yes but slowly.

What I have confirmed works:

• Provisioning the default set of services
• Basic usage of the LMS. It seems to work perfectly fine, just quite a bit more slowly than I'm used to.

I am still in the process of testing this out and setting up my devstack so I will update this with more information.

How it works

• Any services whose docker image already includes ARM support are unchanged (e.g. the MFE node containers, mongo, memcached)
• Any services whose current docker image doesn't support ARM but where there is a compatible image that does support ARM are changed to use the compatible image (e.g. mysql:5.7 → mariadb:10.4, redis:2.8 → redis:5)
• The rest of the containers/services use custom images published by edxops that currently don't support ARM. These ones are just configured to run as x86_64 (amd64) images with qemu emulation. This works fine but is slow.
  • As/if more developers get apple silicon, we should find a way to build multi-arch images for these which will solve the performance problem.

How can I test this?

You need to use either an M1 Mac, an ARM linux system like a Raspberry Pi, or an ARM virtual machine (e.g. use UTM to run arm64 linux on an Intel Mac).

[openedx-webhooks]

Thanks for the pull request, @bradenmacdonald! I've created OSPR-6205 to keep track of it in JIRA, where we prioritize reviews. Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[regisb]

As mentioned on the Open edX forum we are also affected by this issue in Tutor: overhangio/tutor#510

I'm curious to learn what solution will be found for the devstack. In particular, switching to Mariadb seems like a pretty radical choice, given that incompatibilies exist:

https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/
https://mariadb.com/kb/en/incompatibilities-and-feature-differences-between-mariadb-103-and-mysql-57/

(full disclaimer: I'm terrible at understanding mysql but I find this list of incompatibilities long and scary)

[natabene]

@bradenmacdonald Thank you for your contribution. Is this ready for our review?

[bradenmacdonald]

In particular, switching to Mariadb seems like a pretty radical choice, given that incompatibilies exist

Just to be clear, this is only for development, and only for people with ARM systems, and I consider it a temporary solution even for that. I'll certainly update this if I encounter any issues.

[bradenmacdonald]

@natabene It's ready for review yes. I expect there may be some discussion about it, this may or may not be the best direction to go in. But as far as the code goes, I think this is complete.

[bradenmacdonald]

Today I tried building ARM64 images of edxapp but ran into an issue when first trying to build the focal-common base image for ARM64.

cd ..
git clone https://github.com/edx/configuration.git
cd configuration
docker build --platform linux/arm64/v8 -f docker/build/focal-common/Dockerfile --tag edxops/focal-common:latest .

Somehow the task Download digicert intermediate Certificate is failing with a segfault.

<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1636564730.5833266-53209-155692291711061/AnsiballZ_get_url.py && sleep 0'
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "Segmentation fault\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 139
}

Some debugging points to the error is occurring in fetch_url in https://github.com/ansible/ansible/blob/stable-2.8/lib/ansible/module_utils/urls.py

[jinder1s]

I see you are using platform: linux/amd64 in other places in this PR. Have you tried doing that with mysql57, if so, how did it go?

[bradenmacdonald]

I didn't try it, so I'm not sure. Based on my experience with the other images, I suspect it would work fine, but all the database queries would be a bit slower, degrading overall performance.

[msegado]

For what it's worth, the MySQL 8 images published by Oracle support arm64/v8: https://hub.docker.com/r/mysql/mysql-server/tags

[jinder1s]

Hi, thank you for creating this PR. From first glace, it mostly looks good. I added few question in review.

Also, would you either/or modify the README with a note about this or write an ADR about this addition.

[jinder1s]

How much were you able to test this change?

[bradenmacdonald]

I haven't yet done extensive tests. Only basic things like logging into the LMS, using the django admin, and running parts of the test suite. However, so far I haven't encountered any issues.

[jinder1s]

I'm wondering if this should be opt-in. I can't think of why someone with apple silicon would not want to do this, but putting this comment here in case someone reading this can think of a reason.

[bradenmacdonald]

Well the result is pretty slow. But it works, and it's definitely better than not having a devstack at all on M1 Mac.

I am dissatisfied with the performance though, which is why I've now tried building a native ARM image for edxapp instead. If I could get that working, I'd definitely change the PR to use that approach instead. I hit a roadblock, which I think is due to the old version of ansible being used.

[bradenmacdonald]

Also, would you either/or modify the README with a note about this or write an ADR about this addition.

Sure, I just want to spend a bit more time testing this first, and seeing if I can explore an alternative approach.

[msegado]

Looks like the MFEs have an issue too; optipng-bin fails to compile on M1 (see imagemin/optipng-bin#117), which then breaks the build since it's called later in the pipeline.

As suggested in the issue above, one can work around this for now by setting CPPFLAGS=-DPNG_ARM_NEON_OPT=0. I can confirm that this works if added as an environment variable to the MFE services.

[jinder1s]

@bradenmacdonald Thanks again for doing this work. For now, it looks like this is still in-progress. Please ping me when you'd like further review.

[bradenmacdonald]

I'm going to close this PR, and recommend that anyone who wants to develop on an ARM64 device use Tutor devstack instead. See discussion at overhangio/tutor#510 (comment)

Rationale:

• The devstack experience under qemu emulation (which this PR provides) is not great - it's very slow.
• Tutor allows building and running the image natively on arm64, which is fast.
• Debugging the edX ansible scripts to be able to build an arm64 image is painful. It runs ansible stuff for about an hour and then gives a segfault; inspecting the broken container took a lot of work (messing around with "AnsiballZ" internals), but ultimately didn't help me figure out the issue. Debugging the tutor build process is much easier, it builds way faster, and it generally just works.

[openedx-webhooks]

@bradenmacdonald Even though your pull request wasn’t merged, please take a moment to answer a two question survey so we can improve your experience in the future.

[johnnagro]

@bradenmacdonald we are taking another look into M1 support in devstack. Preliminary support on this branch/PR if you're still interested #920

[bradenmacdonald]

@johnnagro Thanks for the heads up. Does that branch run edx-platform in an arm64 container or under x86 emulation? I had been able to get the latter working but not the former.

[johnnagro]

@bradenmacdonald it uses/creates arm containers. we are in the early stages of testing.