Pushing behind traefik is failing

[jeusdi]

I've deployed an registry:2 behind an traefik.

I'm facing with traying to push an image to it:

$ docker push registry.dind.localhost:32785/feedly:v1
The push refers to repository [registry.dind.localhost:32785/feedly]
aa0f3a996547: Preparing 
43c5bd749b73: Preparing 
d7fc70cd5810: Preparing 
27cd2023d60a: Preparing 
4b52dfd1f9d9: Preparing 
error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

I've took a look on dockerd logs:

level=debug msg="Calling POST /v1.40/images/registry.dind.localhost:32785/feedly/push?tag=v1"
level=debug msg="Trying to push registry.dind.localhost:32785/feedly to https://registry.dind.localhost:32785 v2"
level=debug msg="Pushing repository: registry.dind.localhost:32785/feedly:v1"
level=debug msg="Checking for presence of layer sha256:aa0f3a9965479ba5f42cdbdddb14d00fa321d56c267bc0b944c6af8ffbfa7beb (sha256:c602ede608540175f12957c5b9fb7ebeb0e17a9625e0f8e8711306556cb251a8) in registry.dind.localhost:32785/feedly"
level=debug msg="Checking for presence of layer sha256:d7fc70cd581085df5bc3d2ed1249e507b290f9e9873024736a4861870fd00baf (sha256:cf2f38b98af20b81ce03efaa616cda3ff098f68275c55342eceee05f204a326c) in registry.dind.localhost:32785/feedly"
level=debug msg="Checking for presence of layer sha256:4b52dfd1f9d986c4319979f31a32f5ff7acef454f207d2b7ac72bb9f336c6ce8 (sha256:7fe5fcc0340ea9baed3fa3a50c78e997ed51112a739a64f093dd8938656f53c4) in registry.dind.localhost:32785/feedly"
level=debug msg="Checking for presence of layer sha256:43c5bd749b737695c814bb528d170b117e415440a2bd54a27e58d66cb2ecba28 (sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021) in registry.dind.localhost:32785/feedly"
level=debug msg="Checking for presence of layer sha256:27cd2023d60ab891655435a9102be0ce246410f5a54db39edf070a97819b5d38 (sha256:086296bbdfc786dec2bd9f9637f937fcdfaa4ac89de84b99cea50228059aed2c) in registry.dind.localhost:32785/feedly"
level=debug msg="Pushing layer: sha256:aa0f3a9965479ba5f42cdbdddb14d00fa321d56c267bc0b944c6af8ffbfa7beb"
level=debug msg="Pushing layer: sha256:4b52dfd1f9d986c4319979f31a32f5ff7acef454f207d2b7ac72bb9f336c6ce8"
level=debug msg="Pushing layer: sha256:27cd2023d60ab891655435a9102be0ce246410f5a54db39edf070a97819b5d38"
level=debug msg="Pushing layer: sha256:d7fc70cd581085df5bc3d2ed1249e507b290f9e9873024736a4861870fd00baf"
level=debug msg="Pushing layer: sha256:43c5bd749b737695c814bb528d170b117e415440a2bd54a27e58d66cb2ecba28"
level=error msg="Upload failed: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\\n\""
level=error msg="Upload failed: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\\n\""
level=info msg="Attempting next endpoint for push after error: error parsing HTTP 404 response body: invalid character 'p' after top-level value: \"404 page not found\\n\""
level=debug msg="Skipping non-TLS endpoint http://registry.dind.localhost:32785 for host/port that appears to use TLS"

I've also took a look on traefik logs:

"GET /v2/ HTTP/1.1" - - "-" "-" 32 "-" "-" 0ms
"HEAD /v2/feedly/blobs/sha256:c602ede608540175f12957c5b9fb7ebeb0e17a9625e0f8e8711306556cb251a8 HTTP/1.1" - - "-" "-" 33 "-" "-" 0ms
"HEAD /v2/feedly/blobs/sha256:7fe5fcc0340ea9baed3fa3a50c78e997ed51112a739a64f093dd8938656f53c4 HTTP/1.1" - - "-" "-" 34 "-" "-" 0ms
"HEAD /v2/feedly/blobs/sha256:086296bbdfc786dec2bd9f9637f937fcdfaa4ac89de84b99cea50228059aed2c HTTP/1.1" - - "-" "-" 35 "-" "-" 0ms
"HEAD /v2/feedly/blobs/sha256:cf2f38b98af20b81ce03efaa616cda3ff098f68275c55342eceee05f204a326c HTTP/1.1" - - "-" "-" 36 "-" "-" 0ms
"HEAD /v2/feedly/blobs/sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021 HTTP/1.1" - - "-" "-" 37 "-" "-" 0ms
"POST /v2/feedly/blobs/uploads/ HTTP/1.1" - - "-" "-" 38 "-" "-" 0ms
"POST /v2/feedly/blobs/uploads/ HTTP/1.1" - - "-" "-" 39 "-" "-" 0ms
"POST /v2/feedly/blobs/uploads/ HTTP/1.1" - - "-" "-" 40 "-" "-" 0ms
"POST /v2/feedly/blobs/uploads/ HTTP/1.1" - - "-" "-" 41 "-" "-" 0ms
"POST /v2/feedly/blobs/uploads/ HTTP/1.1" - - "-" "-" 42 "-" "-" 0ms

I've also took a look on registry logs and it doesn't generate logs!

Next step: I've tried to reach at registry using curl. I've realized it's reached:

$ curl -s registry.dind.localhost:32785/v2/_catalog | jq
{
  "repositories": []
}

I've also tried to perform previously tried requests by docker push like HEAD _/v2/feedly/blobs/sha256:c602ede608540175f12957c5b9fb7ebeb0e17a9625e0f8e8711306556cb251a8:

$ curl -s -X HEAD -I registry.dind.localhost:32785/v2/feedly/blobs/sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021
HTTP/1.1 404 Not Found

It's also reached and it's telling me layer doesn't exist.

This time, requests has reached to registry. Here registry logs:

level=info msg="response completed" go.version=go1.11.2 http.request.host="registry.dind.localhost:32785" http.request.id=f9698ee1-0ee3-4021-82ab-8b659ffc0549 http.request.method=GET http.request.remoteaddr=10.0.0.2 http.request.uri="/v2/_catalog" http.request.useragent="curl/7.58.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.094913ms http.response.status=200 http.response.written=20 
"GET /v2/_catalog HTTP/1.1" 200 20 "" "curl/7.58.0"
"HEAD /v2/feedly/blobs/sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021 HTTP/1.1" 404 157 "" "curl/7.58.0"
level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="registry.dind.localhost:32785" http.request.id=e42c4aa6-ee13-4eda-ab08-ee15ba3d00a4 http.request.method=HEAD http.request.remoteaddr=10.0.0.2 http.request.uri="/v2/feedly/blobs/sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021" http.request.useragent="curl/7.58.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.692543ms http.response.status=404 http.response.written=157 vars.digest="sha256:0df3b2d147c7a5fefc06f031771af3c9e0905a3e2459f475a3166796b56d1021" vars.name=feedly

Here, you have docker stack with traefik and registry:

version: "3.7"
services:
  traefik:
    image: traefik
    command:
      --api=true
      --api.debug=true
      --api.dashboard=true
      --providers.docker.endpoint=tcp://172.18.0.4:2376
      --providers.docker.tls.ca=/certs/ca.pem
      --providers.docker.tls.cert=/certs/cert.pem
      --providers.docker.tls.key=/certs/key.pem
      --providers.docker.swarmMode=true
      --providers.docker.exposedByDefault=false
      --accesslog=true
    networks:
      - traefik-net
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /certs/client:/certs
    deploy:
      placement:
        constraints: [node.role==manager]
      labels:
        - traefik.enable=true
        - traefik.http.routers.api.rule=Host(`traefik.dind.localhost`)
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.middlewares=auth
        - traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$7h4Zhn7O$$aTpZnWbkA44AWTMgHCwrw1
        # Dummy service for Swarm port detection. The port can be any valid integer value.
        - traefik.http.services.dummy-svc.loadbalancer.server.port=9999
  registry:
    image: registry:2
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.registry.rule=Host(`registry.dind.localhost`) && PathPrefix(`/v2`)
        - traefik.http.services.registry.loadbalancer.server.port=5000
    networks:
      - traefik-net

Any ideas?

[martin2110]

I am hitting this as well. After a bunch of debugging today. I found the following.
They return the same error you pasted:

forwarding port 80 -> 5000 
$ docker push HOSTNAME/iproute2
---
forwarding port 443 -> 5000 
$ docker push HOSTNAME:443/iproute2

However if you use a port that is not 80,443 it works. I tried with 5000, and 444 both work fine.
Hypothesis: docker does something diffrent with 80 and 443.