Add support for multi-arch docker images #399
Conversation
|
|
The version of mambaforge appropriate for the architecture that the docker build is running on is built this way. Ref pangeo-data#396
|
conda-lock also needs to know though. |
|
/condalock |
yuvipanda
changed the title
| - name: Specify which architectures to build lockfiles for | ||
| run: | | ||
| # space delimited list of architectures to send to conda-lock | ||
| echo "BUILD_ARCHS='64 aarch64 ppc64le'" > ${GITHUB_ENV} |
There was a problem hiding this comment.
i think it'd be best to just start with one additional? aarch64 and add others down the line if need be
There was a problem hiding this comment.
@scottyhq yeah i agree I mostly added both to test what packages are needed to be fixed for ppc64le
There was a problem hiding this comment.
Let's build just for linux-aarch64 first as mentioned at #396 (comment)
| conda-lock lock --mamba -k explicit -f environment.yml -p linux-64 | ||
| ../generate-packages-list.py conda-linux-64.lock > packages.txt | ||
| for ARCH in ${BUILD_ARCHS}; do | ||
| conda-lock lock --mamba -k explicit -f environment.yml -p linux-${ARCH} |
There was a problem hiding this comment.
this might be a good time to switch to a unified lockfile and actually add conda-lock to the base environment. (Our current usage of conda-lock pre-dates the unified lockfile format now shown here https://github.com/conda-incubator/conda-lock#basic-usage). Installing the environment with conda-lock would actually recognize the host platform and choose the correct set of packages:
conda-lock lock -f environment.yml -p linux-64 aarch64
conda-lock install -n pangeo-notebook
the drawback is that if someone just has conda or mamba installed they dont recognize the new unified lockfile format... so you could not do conda env create -f conda-lock.yml ...
There was a problem hiding this comment.
@scottyhq ok that's pretty cool and we clearly should just move to that.
There was a problem hiding this comment.
Thinking that we might need to update from conda-lock=1.4 to conda-lock=2.x first at
Also FYI, mamba doesn't support installing from the unified conda-lock.yml, but micromamba does according to https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html#conda-lock-yaml-spec-files. See also mamba-org/mamba#1884 (comment)
| conda-lock lock --mamba -k explicit -f environment.yml -p linux-aarch64; \ | ||
| ../generate-packages-list.py conda-linux-aarch64.lock > packages-aarch64.txt; \ | ||
| conda-lock lock --mamba -k explicit -f environment.yml -p linux-ppc64le; \ | ||
| ../generate-packages-list.py conda-linux-ppc64le.lock > packages-ppc64le.txt; \ |
There was a problem hiding this comment.
see comment above on unified lockfile. but i still really like a simple text file as well though! maybe we create a artifacts subdirectory to organize files not actually used by docker build?
| conda-lock lock --mamba -k explicit -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-aarch64; \ | ||
| ../generate-packages-list.py conda-linux-aarch64.lock > packages-aarch64.txt; \ | ||
| conda-lock lock --mamba -k explicit -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-ppc64le; \ | ||
| ../generate-packages-list.py conda-linux-ppc64le.lock > packages-ppc64le.txt; \ |
There was a problem hiding this comment.
TBH I have rarely used this Makefile and am always just building things via GitHub Actions ;) I've put in a little more time to embracing docker buildx and think it might be the right tool to evolve this repo. in particular the bake tool for several related images https://github.com/scottyhq/pangeo-buildx#usage
There was a problem hiding this comment.
@scottyhq I completely agree - I love that repo and think that's what this should be :D What do you think needs to happen to complete that transition?
There was a problem hiding this comment.
Awesome! Mainly just more time :) I abandoned it for a bit with the "dont fix what isn't broken" rationale :) but it's also nice to keep evolving so I'll try to open a PR here this week and would love your review
Ref #396