Use registry for cache

[jcushman]

These cache flags are wrong:

        BUILD_COMMAND="docker buildx bake -f docker-compose.yml -f docker-compose.override.yml ${{ steps.update-tags.outputs.services-to-rebuild }} \
                      --set *.cache-to=type=local,mode=max,dest=/tmp/.buildx-cache --set *.cache-from=type=local,src=/tmp/.buildx-cache"

The way it wants to work is to cache-from the registry (in all cases), and cache-to the registry (if logged into the registry).

[bensteinberg]

OK, I have what I think is bad news. Although the issues aren't perfectly clear, I believe that the Harbor registry does not recognize the format of the build cache.

goharbor/harbor#14774
goharbor/harbor#16624

This may be true of other registries as well:

As a side note, we use DockerHub as the cache repository instead of Google Container Registry which stores our application images. This is because at the time of writing, Google Container Registry does not seem to support the cache manifest format application/vnd.buildkit.cacheconfig.v0 and returns Bad Request 400 when trying to push a build cache. So we fell back on using a private repo on DockerHub for now and it works perfectly.

(from https://medium.com/titansoft-engineering/docker-build-cache-sharing-on-multi-hosts-with-buildkit-and-buildx-eb8f7005918e)

The evidence here is from running

docker buildx bake -f docker-compose.yml -f docker-compose.override.yml web --push

with a minimal docker-compose file, where docker-compose.override.yml includes

      x-bake:
        tags:
          - registry.lil.tools/harvardlil/caching-test:0.01
        platforms:
          - linux/amd64
          - linux/arm64
        cache-from:
          - type=registry,ref=registry.lil.tools/harvardlil/caching-test:buildcache
        cache-to:
          - type=registry,ref=registry.lil.tools/harvardlil/caching-test:buildcache,mode=max

Unsurprisingly, we see

 => ERROR importing cache manifest from registry.lil.tools/harvardlil/caching-test:buildcache             0.4s

but then at the end we see

 => => writing manifest sha256:d7f3744244341356ea9448a0a46006cd119197af12eac69ad378b6ab54a22317           0.2s
------
> importing cache manifest from registry.lil.tools/harvardlil/caching-test:buildcache:
------
------
> exporting cache:
------
error: failed to solve: error writing manifest blob: failed commit on ref "sha256:d7f3744244341356ea9448a0a46006cd119197af12eac69ad378b6ab54a22317": unexpected status: 404 Not Found

which matches goharbor/harbor#14774.

[bensteinberg]

opencontainers/image-spec#870 has

... There are a few intereseting examples . One is buildkit cache which actually usese descriptors which are blobs and not a oci image manifest. In ACR we went ahead and supported this use case for buildkit.

[bensteinberg]

I believe #8 finally obviates the need for this kind of caching.

[bensteinberg]

Under correction from @rebeccacremona I am reopening this; #8 solves our problem of building and pushing an image too frequently, but does not accomplish the caching of the layers that make up the image. For future reference, a Harbor update might allow us to use a cache manifest, or we might switch to another registry.

[bensteinberg]

750f46aa actually solves the problem of building and pushing too frequently; docker manifest inspect was not working in CI.