tracking issue: chained builds and linuxkit and buildkit #2570
Comments
|
I can't understand how
From my understanding whatever alias we have for I guess we can alias the name to something existing in the docker hub, but that is pointless, because the whole purpose of this efforts is to use something we build localy. |
|
I am going to try and lay out the build issue with a practical example, and then how each potential solution impacts it. Let's say we are a local developer or CI. We are making changes to # first build lfedge/eve-uefi:
$ linuxkit pkg build pkg/uefi
# if the current git tree hash for pkg/uefi is, e.g. 1234567abcd, then
# the above command will create an image - STORED LOCALLY - called "lfedge/eve-uefi:1234567abcd"
# next build lfedge/eve-xen-tools
$ linuxkit pkg build pkg/xen-tools
# if the current git tree hash for pkg/xen-tools is ffbb6677, then the above command will
# create an image - STORED LOCALLY - called "lfedge/eve-xen-tools:ffbb6677"The file FROM lfedge/eve-uefi:1234567abcdFor the second build to work, the builder process must be able to find We recall that The name Because we are in a build process, actually testing changes, that image Looking at the potential solutions. Always PublishThat interim, might be temporary, image, In addition, this only would work for CI, which has credentials to push to Docker cacheIn a "normal" FROM lfedge/eve-uefi:1234567abcdis interpreted as: "first look for This is useful behaviour for builds, and nearly every docker-based process depends upon it for the same reasons we do: build locally, Unfortunately, this does not work with container-driver buildkit, which is the source of all of our problems. There is no way to tell container-driver buildkit, "go look in this cache before going to a registry". Local RegistryIn this case, even though the image name in the line is: FROM lfedge/eve-uefi:1234567abcdwhich means, "use the image For this to work, the builder would need to know, "do not go to docker hub, but rather go to some local registry at ARG uefi=lfedge/eve-uefi:1234567abcd
FROM ${uefi}A normal build of The challenges are:
This solution should work, if somewhat messy. buildkit image cachingI have wanted this for nearly a year. The same way docker has, "I first will look in my local cache, and only go registry if I cannot find it", I want buildkit to support the same thing. This is the best solution, both technically and operationally, but we are waging an uphill battle to make it happen. buildkit named contextsNamed contexts essentially let you alias an image on a FROM lfedge/eve-uefi:1234567abcdwe could run That some other source could be:
For this to work, we would need:
buildkit bake contextsThese are just fancier versions of buildkit build contexts. Rather than specify it on the command-line with For example, we would do: target "xen-tools" {
contexts = {
"lfedge/eve-uefi:1234567abcd" = "<some other source>"
}
}This then becomes the same question: where is |
|
bake-file is still required to publish to the registry or cache. This is a minimal example and commands: |
|
You don't need a bakefile publish. Someone had a good idea on using bake files, but only when dealing with local cache. I might try that, and if it works, get it into linuxkit. It still becomes a headache. You would need to know at building the first image that you should "extra cache" it, and at the second image that you should source from that cache. What does that CLI look like? Something like that? This is ugly. |
|
I may have a better solution. Still not perfect, but it looks better. Recall the issue:
As described above, buildkit has the ability to use build contexts (using If we add oci layout as a place to get it - which they have agreed with - then we could tell it, "go find The process then would look like: lkt pkg build A # unchanged
lkt pkg build B --from-cache A # last flag added, meaning, "got get A from my linuxkit cache"I am working on it. Once I fully understand the way to add a source, I will submit a PR to buildkit. It is a bit (a lot) challenging to grasp. |
@deitch Can there be fallback so that we don't need to track whether A is in the build cache i.e., have it look in the cache first and if not found also look in registry? |
|
I am +1 on @eriknordmark suggestion to the last @deitch idea -- @deitch what would it take to implement this in lkt? |
|
Yeah, I think it can. To recap:
|
|
Great @deitch ! Btw, what about cross builds in buildkit -- with what you're working on -- will we have that for free or is it still separate work? |
|
Depends what you mean by cross builds. If you mean, "build for amd64 whole on arm64 using emulation", then, yeah, lkt already does it. It also can inherit remote builders for quicker builds. We just don't take advantage of it because of that issue. This will inherit all of that. If you mean literal cross compiling, as in native go on amd64 compiling for arm64, it doesn't do that yet. I could get us there, but it is a lift. |
Yeah -- that's what I'm after -- can you please keep that usecase in mind while you're working on this issue? I can help test/etc. too -- really hope we can restructure our Dockerfiles/lkt/buildx setup in such a way that it would simply work out of the box. If not -- we may need to file additional issues I guess -- but that that kind of investigation is what I'm after. |
|
The first buildkit PR is merged in!! The first linuxkit PR is open and CI is green. More coming. |
|
Linuxkit PR is merged in. Linuxkit now executes buildkit directly. One more PR for linuxkit to support the cached build, and it is done. |
This is a tracking issue for the chained builds problem. It is intended to be used as a place to track the issue and discuss possible solutions.
cc the following people who have been helping look into it: @eriknordmark @rvs @petr-zededa @ruslan-zededa . With many thanks to them.
Problem Description
Dockerfiles. Each of these consumes other images inFROM, sometimes multiple.FROMimages via networked registries, i.e.FROM eve/foo:1.2.3will only look in dockerhub fromeve/foo:1.2.3. It does not look in the local cache, whether linuxkit's or docker's.The above is the reason why the eve-os build process does not use the latest linuxkit. We cannot combine the docker-style "build an image, save it locally, build another that consumes that locally saved image" with the buildkit-style "build images across architectures". This is the catch-22
Scope
This only affects
DockerfiletoDockerfile"chained" builds. This does not affect final OS image build. The outputs of the variousdocker buildcommands is consumable by linuxkit wherever it is.Impacted Files
In order to get a better handle on the scope of the problem, I ran a grep through all of lf-edge/eve to find what files are affected. Excluded vendor directories and our unique
.gocaching system.the results show 85
FROMinDockerfiles.Next, I excluded any that will always or never come from a registry, primarily upstream
alpine,golang,unikernel,scratch, simplifying it further:The results are down to 45 files.
Now we just need to see the value of
FROMand how many times it is used:the results:
The
EVE_BUILDER_IMAGEis set tolfedge/eve-alpine:6.7.0in all cases, it just uses it as a build arg viaARG EVE_BUILDER_IMAGE=lfedge/eve-alpine:6.7.0. So we can redo the results as:The problem arises when we want to build any one of those - primarily
lfedge/eve-alpine- locally, and check its value by using it downstream. For example, if we make changes tolfedge/eve-alpine, call itlfedge/eve-alpine:6.8.0-test, and want to use it in other pkgs, the build would fail, as it is not pushed out to docker hub, nor should it be.In cases where we are just building the downstream packages that consume a well-known and published
lfedge/eve-alpine, it will not be an issue, as it is downstream.The same is true for the interpolated value of
MKISO_TAGandMKCONF_TAG, etc. If we are building just the downstream packages, then the current value interpolated will already have been published. But if we are building the source ofMKISO, and it has not been published yet, then the downstream build will fail.potential solutions
This section is for suggestions as to how to fix it.
Some starting points, none of which is necessarily good.
localhost:8800/lf-edge/eve-alpine:7.6.0or similar in the dockerfile. This might work if we set all of them to use build args, but it is messy and requires a lot of extra overhead. It also is not clear that linuxkit supports build-args (although those could be added).Looking forward to insights.
The text was updated successfully, but these errors were encountered: