-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide linux/arm64 architecture support with an open-source chain of images (eg. forem/ruby) #19626
Comments
Thanks for the issue, we will take it into consideration! Our team of engineers is busy working on many types of features, please give us time to get back to you. To our amazing contributors: issues labeled If this is a feature request from an external contributor (not core team at Forem), please close the issue and re-post via GitHub Discussions. To claim an issue to work on, please leave a comment. If you've claimed the issue and need help, please ping @forem-team. The OSS Community Manager or the engineers on OSS rotation will follow up. For full info on how to contribute, please check out our contributors guide. |
It has been way too long since I've worked on this lol. Didn't expect this to become a real thing xD. |
Ok I'm working on this again because I want this too and I want to help :) |
Hey @RedstoneWizard08, thanks for the draft pull request and for reviving the branch - we really appreciate the enthusiasm! That said, we've already started this work internally and are doing last passes of testing on (1) a fully revised image that doesn't use Fedora as a base at all (making this draft hard to reconcile with the work ongoing), and (2) should come with ARM64 support out of the box. In general, we attach a Again, thanks so much for opening a draft with the branch that held that work you'd already tackled, but no need to stress yourself (or burn your nights and weekends, or whatever) on this any further - I'm tackling this one as part of my day job here at Forem :) For future such issues, look for that |
Okay, I didn't realize that's what that meant. |
This builds a Debian-derived base image that targets Ruby 3.0.2 (our current version lock) to replace `quay.io/forem/ruby`, which is based on Fedora 35 (and as such, lacks ARM64 support). The building of this image is not yet automated, I intend to add that as a follow-up some time vaguely soon-ish. This has been built and pushed to a temporary tag, [ghcr.io/forem/ruby:klardotsh-test](https://github.com/orgs/forem/packages/container/ruby/103882793?tag=klardotsh-test). Once this merges, I'll build and push the result in main (pending any revisions that come up in PR review) with the following incantation, which will generate a multiarch manifest: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile.base . -t ghcr.io/forem/ruby:latest -t ghcr.io/forem/ruby:$(git rev-parse --short HEAD) --push ``` This refs, but does not complete, #19626, and is one of several blockers on the path to getting #19603 merged.
This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
…e. (#19632) This builds a Debian-derived base image that targets Ruby 3.0.2 (our current version lock) to replace `quay.io/forem/ruby`, which is based on Fedora 35 (and as such, lacks ARM64 support). The building of this image is not yet automated, I intend to add that as a follow-up some time vaguely soon-ish. This has been built and pushed to a temporary tag, [ghcr.io/forem/ruby:klardotsh-test](https://github.com/orgs/forem/packages/container/ruby/103882793?tag=klardotsh-test). Once this merges, I'll build and push the result in main (pending any revisions that come up in PR review) with the following incantation, which will generate a multiarch manifest: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile.base . -t ghcr.io/forem/ruby:latest -t ghcr.io/forem/ruby:$(git rev-parse --short HEAD) --push ``` This refs, but does not complete, #19626, and is one of several blockers on the path to getting #19603 merged.
This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
…e. (#19632) This builds a Debian-derived base image that targets Ruby 3.0.2 (our current version lock) to replace `quay.io/forem/ruby`, which is based on Fedora 35 (and as such, lacks ARM64 support). The building of this image is not yet automated, I intend to add that as a follow-up some time vaguely soon-ish. This has been built and pushed to a temporary tag, [ghcr.io/forem/ruby:klardotsh-test](https://github.com/orgs/forem/packages/container/ruby/103882793?tag=klardotsh-test). Once this merges, I'll build and push the result in main (pending any revisions that come up in PR review) with the following incantation, which will generate a multiarch manifest: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile.base . -t ghcr.io/forem/ruby:latest -t ghcr.io/forem/ruby:$(git rev-parse --short HEAD) --push ``` This refs, but does not complete, #19626, and is one of several blockers on the path to getting #19603 merged.
…#19633) This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
…#19633) This "rebases" our application images off of the new Ruby base image added in #19632, and fixes numerous problems and quirks with how the images were built along the way. Notably: - Issues where layers attempted to delete files in prior layers have been resolved (this caused build failures on some Docker filesystem drivers, notably overlay2). - Bundler is no longer allowed to deviate from or modify the lockfile (`BUNDLE_FROZEN` is now `true`). - `git(1)` is no longer required to live inside the container and `.git` is no longer required to be copied into the Docker build context, as these were only used to calculate `FOREM_BUILD_SHA`, which is now passed in as a Build Argument to the container build context. - The entire source tree is no longer `chmod` in one giant swing, which ran so long on my system (as just one example) that I gave up after 15-20 minutes and issued it a `SIGTERM`. Instead, `COPY --chown` is used more heavily and ensures the `APP_USER` will have access to the requisite files. This new container image appears to build successfully for `linux/arm64`, which refs (but does not complete) #19626. Currently, such builds aren't automated , and must be built on a developer workstation. For example: ```sh docker buildx build --platform linux/amd64,linux/arm64 -f Containerfile . -t ghcr.io/forem/forem:klardotsh-test --push --build-arg VCS_REF=$(git rev-parse --short HEAD) ``` In the meantime, the existing `linux/amd64`-only BuildKite scripts have been updated to allow this PR to merge as a separate unit, and CI refactors to enable the multiarch builds of `linux/arm64,linux/amd64` can come later when more time is available. This is one of several blockers on the path to getting #19603 merged. The next step in that chronology will be rebasing that work on top of this work, which *should* be, on the containerization side, as straightforward as bumping `Containerfile.base` to reference the new upstream image, rebuilding the base container, and then bumping the reference in `Containerfile`.
I think this is going to be tricky to deal with: on an AMD 6900HX with 32GB of RAM, the AMD64 images build in ~10 minutes or so (maybe 15?), but I just clocked in about 2 hours for the total multiarch manifest counting the QEMU-based ARM64 cross compile. I think actually shipping ARM64 images as part of our CI pipeline is going to require setting up GitHub Actions or BuildKite agents on actual ARM64 instances (perhaps AWS Graviton2), building the image natively on each arch, and then stitching the two images together with a custom manifest in a third CI step. This is a noteworthy bit of infra tinkering to take on, and so I don't think this issue is likely to close fully within the next month: I'll work with @mirie to slot this build infra work in as our bandwidth allows. In the mean time, I feel reasonably confident in someone's ability to |
This looks amazing so far! If I may suggest, although I don't know if it has been tried yet, I was able to mitigate the speed problem to some degree using GitHub actions caching on docker buildx. |
Discussed in #16765
Originally posted by RedstoneWizard08 March 2, 2022
Description
Provide support for arm64 CPUs to run the Docker image of Forem. This will include the Mac M1 chips, general aarch64 chips (
linux/arm64
), and Raspberry Pis. This will also continue to allow amd64 chips to run it as well.Potential Steps
Update Forem ruby image for arm64 and amd64 (and please make it open-source), Dockerfile is below.
Possible Forem AIO image (but use the regular one, it's better, just add arm64 support)
Update Forem docker image to allow for arm64 and amd64 architectures.
Notes
RUN export ARCH=$(uname -m)
Files
See here: https://github.com/RedstoneWizard08/Forem-Multiarch-Docker
The text was updated successfully, but these errors were encountered: