Fix manifest list never being stored in proxy-cache projects

[cnmcavoy]

Fix manifest list never being stored in proxy-cache projects by looking up the manifest on HEAD requests

Comprehensive Summary of your change

My understanding is that when docker (or containerd) requests a manifest list in a proxy-cache project, the client never makes a GET request for the manifest list. It makes a HEAD request for the manifest list, then makes a GET request for the platform architecture specific digest. As a result, the manifest list is never persisted in the storage layer, only in the redis cache layer. When the cache expires, this forces a new remote lookup for future requests instead of being able to rely on the storage, which is the purpose of the proxy-cache.

Issue being fixed

This may or may not also fix #19609, but it does seem related.

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

[Vad1mo]

Interesting observation, and hard to spot edge case. Do you have a step-by-step guide on how to reproduce the bug?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 67.45%. Comparing base (b7b8847) to head (240af51).
Report is 88 commits behind head on main.

❗ Current head 240af51 differs from pull request most recent head c859215. Consider uploading reports for the commit c859215 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #19910      +/-   ##
==========================================
- Coverage   67.56%   67.45%   -0.11%     
==========================================
  Files         991      996       +5     
  Lines      109181   109777     +596     
  Branches     2719     2720       +1     
==========================================
+ Hits        73768    74051     +283     
- Misses      31449    31744     +295     
- Partials     3964     3982      +18     

Flag	Coverage Δ
unittests	67.45% <0.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/server/middleware/repoproxy/proxy.go	2.65% <0.00%> (-0.05%)	⬇️

... and 23 files with indirect coverage changes

[cnmcavoy]

Interesting observation, and hard to spot edge case. Do you have a step-by-step guide on how to reproduce the bug?

We detected this because we have a Harbor / Docker-registry setup that is sensitive the filesystem layout that Harbor owns. We have a primary Harbor that runs in one primary kubernetes cluster. We configure Harbor to use S3 as the filesystem, and we have other clusters which run docker-registry's which are given limited read access out of that S3 bucket. Those docker-registry can read Harbor's filesystem, because both are the same fundamental component, and so registries are affected by the filesystem, rather than the contents of Harbor's middleware (redis, in our case). In our case, we detected that manifest-list images were only partially mirrored, that the manifest layers pointed to by the manifests in the manifest-list were copied into the S3 bucket, but the manifest-list itself was absent.

To give a concrete example, if I inspect the current ubuntu focal manifest-list, it returns something like this:

{
  "manifests": [
    {
      "digest": "sha256:48c35f3de33487442af224ed4aabac19fd9bfbd91ee90e9471d412706b20ba73",
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 424
    },
...
}

When we made requests to Harbor it would have no problem proxying the ubuntu:focal manifest-list and containers to the client, and the S3 bucket would contain the architecture of the image we requested, for example, the amd arch and "sha256:48c35f3de33487442af224ed4aabac19fd9bfbd91ee90e9471d412706b20ba73". This is expected, but what was missing is that the manifest-list wasn't stored. So pulling directly from the S3 bucket failed, but pulling through Harbor succeeded, even though Harbor was replying with layers that pointed at the S3 bucket. Harbor was caching the container layers, but not the manifest-list.

This confused me initially because when I read the Harbor proxy cache code, the code seems to explicitly handle manifest-lists correctly. But when I inspected the layers in the docker-registry after pulling a manifest-list image for the first time through the proxy cache, you will find the layers for the inner container, but not the manifest list or any of the file pointers created for that resource. And the reason I found for this is that the docker client at some point in the past, used to make a request flow for an image that looked something like this: client -> HEAD $MANIFEST, client -> GET $MANIFEST, client fetches the layers. And when docker added manifest-lists, they added a step of indirection on the client side. So the client -> HEAD $MANIFESTLIST, then client GET -> $MANIFEST, and never executes a get on a manifest list resource directly. The client uses the list to pick out it's arch specific manifest and never refers to the list again. Harbor never sees the GET request on the manifest-list, and never saves the manifest list to disk.