A path to supporting ARM

[acj]

Hello, all. There seems to be demand for ARM support in Lilliput, and several folks have proposed how to do it, but it's not clear what the project maintainers would be willing to accept contribution-wise. My hope is to gather the relevant threads here and to start that conversation.

Most of the friction is related to supporting Apple Silicon so that developers can build Lilliput on modern Mac hardware. There's also friction, though perhaps less, related to building for arm64 on Linux to be deployed in production environments. There's a general expectation that packages support both x86_64 and arm64 these days.

I see three main problems to navigate:

1. Deciding which combinations of OS and architecture to support. A maintainer has suggested that the x86 deps for macOS might not be worth providing anymore. The author of Add ARM support #98 mentioned that including deps for {darwin,linux}x{x86_64,arm64} approaches the 500MB limit for Go modules. Supporting linux/x86_64 seems non-negotiable. What else should be included?
2. Updating the cgo directives in {avcodec,giflib,opencv,thumbhash,webp}.go to support multiple architectures. This is straightforward (example: Add ARM support #98) once we know the OSes and architectures that Lilliput should support.
3. It's not clear how the pre-compiled dependencies are actually built and packaged in Discord's CI system, which makes it difficult for community devs to build Lilliput for environments that aren't (yet) supported. It's also difficult to do a supply chain audit of the dependency build, which is increasingly important to many developers. While we don't need to resolve this issue in order to support ARM, I think it would be very helpful.

I've made a prototype branch to make clear what I'm proposing, specifically this commit for the CI workflow and this one for adding support for Apple Silicon. The rest of the commits address various build issues that I ran into. There are sample CI runs for building dependencies and building and testing the package.

Questions for maintainers

1. Would you accept a PR that adds support for macOS on ARM and drops support for macOS on Intel? If not, what combination would you prefer to support?
2. Would you welcome a public-facing CI workflow that uses GitHub Actions?

@brian-armstrong-discord @skidder @walruscow (recent committers 👋)

Related: #95 #98 #111 #138 #139

[skidder]

@acj Thank you for the detailed issue and your interest in contributing to the project!

1. Would you accept a PR that adds support for macOS on ARM and drops support for macOS on Intel? If not, what combination would you prefer to support?

We have no uses for the MacOS Intel libraries at this time. All of our development and production uses of Lilliput run on Linux x86. So, the lagging MacOS dependencies, whether Intel or ARM, has not been a significant concern for us.

Nonetheless, I understand that others in the community might rely on this MacOS dependencies.

I'd happily review a PR with the changes you described!

2. Would you welcome a public-facing CI workflow that uses GitHub Actions?

Can you tell me more? Again, happy to review and talk over a PR! :)

Thanks again!
--Scott

[acj]

Thanks, Scott! That context is very helpful.

I'd happily review a PR with the changes you described!

👍 I'll get it ready.

Would you welcome a public-facing CI workflow that uses GitHub Actions?

Can you tell me more?

Yes, this relates to problem 3 above. The gist is that a GitHub Actions workflow like this prototype would build and test Lilliput on each commit. (GHA is free for open source repos.) The prototype also builds all of the dependencies and uploads them as a tarball for easy local testing. The goal is to give community members a way to verify their changes before/after making a PR, and to demonstrate how we expect the build process to work on supported platforms.