Cannot login to docker 400 Bad Request

[jonsch318]

Hello,

In the last week i tried deploying harbor via helm ready to be in HA (not yet: still replica=1).

Until now i have finally been able to access the harbor ui and create a user.
But now when i try to docker login registry.[tldomain] i get prompted to enter username & password and after that:

Error response from daemon: login attempt to https://registry.[tldomain]:443/v2/ failed with status: 400 Bad Request

Harbor environment: Many different configs tried
Version v2.10.0-6abb4eab
Traefik ingress with custom IngressRoute

internalTLS: enabled (when disabling it does not work either) using custom certs provided by certmanger.
postgres: external (no logs indicate a failure, all permissions granted)
redis: generated looks good
core:
secret: given as existingSecret 16 chars via vault-secrets-operator
xsrf & token tls: generated using helm
jobservice:
secret: given as existingSecret 16 chars via vault-secrets-operator
registry:
secret: given as existingSecret 16 chars via vault-secrets-operator
trivy: disabled

When logging in with docker login -u [username] --password-stdin registry.[tldomain] docker never prompts for a password
When logging in with docker login registry.[tldomain] i get prompted to enter username&password and then

Error response from daemon: login attempt to https://registry.[tldomain]:443/v2/ failed with status: 400 Bad Request

During this deployment/registry-harbor-core does not log anything.
Traefik access logs:

[traefik ip]  - - [18/Feb/2024:23:07:57 +0000] "GET /v2/ HTTP/1.1" 401 76 "-" "docker/25.0.2 go/go1.21.6 git-commit/fce6e0ca9b kernel/6.7.4-arch1-1 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.2 \(linux\))" 296 "registry-[ingressname]@kubernetescrd" "https://[internal core ip]:8443" 130ms

[traefik ip]  - - [18/Feb/2024:23:07:58 +0000] "GET /service/token?account=[username]&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 200 633 "-" "docker/25.0.2 go/go1.21.6 git-commit/fce6e0ca9b kernel/6.7.4-arch1-1 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.2 \(linux\))" 297 "registry-[ingressname]@kubernetescrd" "https://[internal core ip]:8443" 23ms

[traefik ip] - - [18/Feb/2024:23:07:58 +0000] "GET /v2/ HTTP/1.1" 400 0 "-" "docker/25.0.2 go/go1.21.6 git-commit/fce6e0ca9b kernel/6.7.4-arch1-1 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.2 \(linux\))" 298 "registry-[ingressname]@kubernetescrd" "https://[internal core ip]:8443" 20ms

When giving an invalid password

2024-02-18T22:57:57Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="[traefik ip]" requestID="1888ce82-46d5-4549-906a-bccab683a797" user agent="docker/25.0.2 go/go1.21.6 git-commit/fce6e0ca9b kernel/6.7.4-arch1-1 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.2 \(linux\))"]: failed to authenticate user:[username], error:Failed to authenticate user, due to error 'Invalid credentials'

So i know i did not enter the wrong credentials. I it must have something to do with the ingress and http headers.

I'm trying to get more information about the http headers.
This does also not work with podman.

Has anyone encountered a similar problem?

[jonsch318]

I did some further digging:

Via /service/token?account=[username]&offline_token=true&service=harbor-registry&client_id=docker

i can get a token.
When queriying /v2/ With the given Authorization: Bearer [token] Header and the csrf token. A 400 Bad Request results. I don't know the registry v2 api but from jaeger tracing i can see that docker uses /v2/ to verify successfull login.

The logs during this on harbor-core:

// 401 find bearer auth realm
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id cbf5eb10-ba5f-43d3-ac7c-e8f14c05eff7 to the logger for the request GET /v2/
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="cbf5eb10-ba5f-43d3-ac7c-e8f14c05eff7" traceID="3c2f850276b3bb4feb03d02f5322a427"]: an unauthorized security context generated for request GET /v2/
2024-02-19T18:35:44Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}

//Get service token
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 65a244d4-85e8-4d1e-aee8-cb0771ff7a15 to the logger for the request GET /service/token
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?account=[username]&client_id=docker&offline_token=true&service=harbor-registry
2024-02-19T18:35:44Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2024-02-19T18:35:44Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="65a244d4-85e8-4d1e-aee8-cb0771ff7a15" traceID="002c36e7a6c18c43df084ddcfd6702c8"]: a basic auth security context generated for request GET /service/token
2024-02-19T18:35:44Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?account=[username]&client_id=docker&offline_token=true&service=harbor-registry
2024-02-19T18:35:44Z [DEBUG] [/core/service/token/creator.go:201]: scopes: []
2024-02-19T18:35:44Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: []

//FAILED /v2/ 400 BadRequest
2024-02-19T18:35:45Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 7bda76f9-3bd2-42bf-9d9e-2cbee09f1156 to the logger for the request GET /v2/
2024-02-19T18:35:45Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/

[TheMatrix97]

Same error here. I'm my case I'm able to run helm repository login without any issue, but docker login throws error 400...
I'm running a harbor in Minikube accessing via port forward to 8443, just as a proof of concept

[jonsch318]

Same error here. I'm my case I'm able to run helm repository login without any issue, but docker login throws error 400...

I never tested it until now. But in my case it fails as well:

INFO[0012] Error logging in to endpoint, trying next endpoint  error="login attempt to https://registry.[tldomain]/v2/ failed with status: 400 Bad Request"
Error: login attempt to https://registry.[tldomain]/v2/ failed with status: 400 Bad Request

[zyyw]

Hi @jonsch318 , Traefik ingress is not tested. Have you got a chance to try nginx ingress?
BTW, there is a known issue of harbor-helm v1.14.0 (although this issue may not related to the 400 error), please try it on v1.13.2 for now.

[zyyw]

also, please check this issue #582 for information of using Traefik with Harbor-helm

[jonsch318]

Unfortunately I am stuck with traefik. But it shouldn't matter since all Authorization headers get forwarded successfully (since it would say 401 otherwise i think).

In version v1.13.2 the issue still persists

[TheMatrix97]

I switched from Node-Port mode to Ingress with Nginx Ingress Controller and It worked fine.... I think the issue has to be related somehow with the FQDN domain rule matching at nginx level But I can't confirm that.... El sáb, 24 feb 2024 a las 12:45, Jonas Schneider ***@***.***>) escribió:

…

Unfortunately I am stuck with traefik. But it shouldn't matter since all Authorization headers get forwarded successfully (since it would say 401 otherwise i think). In version v1.13.2 the issue still persists — Reply to this email directly, view it on GitHub <#1710 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLBWYX26KOLAO24DUZCKRDYVHGRHAVCNFSM6AAAAABDOQ2BQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRSGMZTKOJSGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[jonsch318]

unfortunately even with a nginx intermediate proxy like @TheMatrix97 it is not working. As well as when i use nginx as deployed in cluster ip expose mode and just port-forward to it.

It is a underlying issue of harbor core as i think

[jonsch318]

I just updated to 1.14.1 helm chart and Harbor 2.10 and the problem still persists

[yodatak]

I got the same issue with latest helm chart Harbor 2.10 and traefik

[dgioulakis]

Just installed Harbor and now seeing this same issue.