Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

On tls issues with the certificate - token service returns empty token without any error or log record #20080

Open
dee-kryvenko opened this issue Mar 3, 2024 · 5 comments · May be fixed by #20081
Open

On tls issues with the certificate - token service returns empty token without any error or log record #20080

dee-kryvenko opened this issue Mar 3, 2024 · 5 comments · May be fixed by #20081
Assignees
@MinerYang
Labels
candidate/2.12.0

Comments

@dee-kryvenko
Copy link

dee-kryvenko commented Mar 3, 2024 •
edited

Expected behavior and actual behavior:
In my case, certificate I was using seems not to be compatible with current Harbor code (which is a separate issue).

If I were to pull an image:

# crictl --debug pull harbor.example.com/hub.docker.com/jenkins/jenkins:lts
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:harbor.example.com/hub.docker.com/jenkins/jenkins:lts,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
E0303 20:16:54.430697 1755705 remote_image.go:167] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"harbor.example.com/hub.docker.com/jenkins/jenkins:lts\": failed to resolve reference \"harbor.example.com/hub.docker.com/jenkins/jenkins:lts\": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response" image="harbor.example.com/hub.docker.com/jenkins/jenkins:lts"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to resolve reference "harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response

With corresponding logs on the Core side:

2024-03-03T20:18:41Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?scope=repository%3Ahub.docker.com%2Fjenkins%2Fjenkins%3Apull&service=harbor-registry
2024-03-03T20:18:41Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="1fe703fb-ffa5-4fdc-a527-9a25e6361801"]: an unauthorized security context generated for request GET /service/token
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?scope=repository%3Ahub.docker.com%2Fjenkins%2Fjenkins%3Apull&service=harbor-registry
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/creator.go:201]: scopes: [repository:hub.docker.com/jenkins/jenkins:pull]
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:hub.docker.com/jenkins/jenkins:pull]
2024/03/03 20:18:41 Model:
2024/03/03 20:18:41 r.r: sub, obj, act
2024/03/03 20:18:41 p.p: sub, obj, act, eft
2024/03/03 20:18:41 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2024/03/03 20:18:41 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2024/03/03 20:18:41 g.g: _, _
2024/03/03 20:18:41 Policy:
2024/03/03 20:18:41 p: sub, obj, act, eft: [[anonymous /project/2 read allow] [anonymous /project/2/label read allow] [anonymous /project/2/label list allow] [anonymous /project/2/repository list allow] [anonymous /project/2/repository pull allow] [anonymous /project/2/scan read allow] [anonymous /project/2/scanner read allow] [anonymous /project/2/tag list allow] [anonymous /project/2/artifact read allow] [anonymous /project/2/artifact list allow] [anonymous /project/2/artifact-addition read allow] [anonymous /project/2/accessory list allow]]
2024/03/03 20:18:41 g: _, _: []
2024/03/03 20:18:41 Role links for: g
2024/03/03 20:18:41
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, delete ---> false
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, scanner-pull ---> false
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, pull ---> true
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  hub.docker.com/jenkins/jenkins [pull]}

Steps to reproduce the problem:
I recon there are multiple ways, but in my case I was using a following CA:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-tls-ca
spec:
  isCA: true
  commonName: internal-tls-ca
  secretName: internal-tls-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: self-signed
    kind: ClusterIssuer
    group: cert-manager.io

I'm guessing ECDSA algorithm is not currently supported.

Versions:
Please specify the versions of following systems.

  • harbor version: [v2.10.0]
  • harbor chart version: [v1.14.0]

Additional context:

CA secret from is given to .core.secretName.

Thread on slack: https://cloud-native.slack.com/archives/CC1E09J6S/p1709497679881159
This was referenced Mar 3, 2024
Closed
Open
Open
Open
@MinerYang
Copy link
Contributor

MinerYang commented Mar 4, 2024 •
edited

from the debug msg in harbor-core, the user is empty ""

[DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  hub.docker.com/jenkins/jenkins [pull]}

thus the returned token is actually unauthoirzed to your repository. Have your properly save your credential file when using crictl?

harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response

Could you provide your full service token request when pulling?

@dee-kryvenko
Copy link
Author

This is a public repository and there is no user. I am trying to pull as anonymous. In slack, I left some more logs on what happens when I try nerdctl login, it is not any less ambiguous error stemming from the same problem.

@dee-kryvenko
Copy link
Author

Plus the log literally tells you Request: anonymous, /project/2/repository, pull ---> true

@github-actions GitHub Actions
Copy link

github-actions bot commented May 4, 2024

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label May 4, 2024
@dee-kryvenko
Copy link
Author

Not stale

@MinerYang MinerYang self-assigned this May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MinerYang MinerYang

Labels
candidate/2.12.0
Projects
None yet
Milestone
No milestone
2 participants
@dee-kryvenko @MinerYang