Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Harbor Docker Login 401 behind OPNSense #1743

Open
gsaudade99 opened this issue Apr 16, 2024 · 1 comment
Open

Harbor Docker Login 401 behind OPNSense #1743

gsaudade99 opened this issue Apr 16, 2024 · 1 comment
Assignees
@wy65701436 @zyyw
Labels
help wanted Extra attention is needed

Comments

@gsaudade99
Copy link

Hello, Im currently testing k8s harbor setup in our new datacenter. The datacenter is behind a opnsense. I have managed to migrate services and resolve them using the HAProxy in the opnsense.

I have managed to install harbor, login and resolve the dns correctly. Harbor is exposed using metallb in l2 mode.
Im currently facing the issue where It keeps retrying to push the image to the repository and eventually gives a 500 erorr.

All ssl termination is in the HAProxy.

➜  harbor docker login harbordc.ubiwhere.com -u admin
Password: 
Login Succeeded

➜  harbor time docker push harbordc.ubiwhere.com/test-speed/1gb-random-file:latest

The push refers to repository [harbordc.ubiwhere.com/test-speed/1gb-random-file]
613d42ba11ea: Pushing [==================================================>]  1.074GB
e154057080f4: Pushing [==================================================>]   4.23MB
received unexpected HTTP status: 500 writing request for harbordc.ubiwhere.com:80: write tcp 10.255.2.3:43868->91.209.16.33:80: write: broken pipe
docker push harbordc.ubiwhere.com/test-speed/1gb-random-file:latest  0.15s user 0.09s system 0% cpu 3:19.58 total

Bellow Is my values.yml configuration:

expose:
  type: loadBalancer
  tls:
    enabled: false
    certSource: auto
    auto:
      commonName: "harbordc.ubiwhere.com"
    secret:
      secretName: ""
  ingress:
    hosts:
      core: harbordc.ubiwhere.com
    controller: default
    kubeVersionOverride: ""
    className: ""
    annotations:
      ingress.kubernetes.io/ssl-redirect: "true"
      ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
    harbor:
      annotations: {}
      labels: {}
  clusterIP:
    name: harbordc
    staticClusterIP: ""
    annotations: {}
    ports:
      httpPort: 80
      httpsPort: 443
  nodePort:
    name: harbordc
    ports:
      http:
        port: 80
        nodePort: 30002
      https:
        port: 443
        nodePort: 30003
  loadBalancer:
    name: harbordc
    IP: "<some_ip>"
    ports:
      httpPort: 80
      httpsPort: 443
    annotations: {}
    sourceRanges: []

externalURL: https://harbordc.ubiwhere.com

internalTLS:
  enabled: false
  strong_ssl_ciphers: false
  certSource: "auto"
  trustCa: ""
  core:
    secretName: ""
    crt: ""
    key: ""
  jobservice:
    secretName: ""
    crt: ""
    key: ""
  registry:
    secretName: ""
    crt: ""
    key: ""
  portal:
    secretName: ""
    crt: ""
    key: ""
  trivy:
    secretName: ""
    crt: ""
    key: ""

ipFamily:
  ipv6:
    enabled: true
  ipv4:
    enabled: true

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: ""
        storageClass: "longhorn"
        subPath: ""
        accessMode: ReadWriteOnce
        size: 1Gi
        annotations: {}
    database:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    redis:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    trivy:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
  imageChartStorage:
    disableredirect: false

    type: filesystem
    filesystem:
      rootdirectory: /storage
    azure:
      accountname: accountname
      accountkey: base64encodedaccountkey
      container: containername
      existingSecret: ""
    gcs:
      bucket: bucketname
      encodedkey: base64-encoded-json-key-file
      existingSecret: ""
      useWorkloadIdentity: false
    s3:
      region: us-west-1
      bucket: bucketname
    swift:
      authurl: https://storage.myprovider.com/v3/auth
      username: username
      password: password
      container: containername
      existingSecret: ""
    oss:
      accesskeyid: accesskeyid
      accesskeysecret: accesskeysecret
      region: regionname
      bucket: bucketname
      existingSecret: ""

imagePullPolicy: IfNotPresent

imagePullSecrets:

updateStrategy:
  type: RollingUpdate

logLevel: info

existingSecretAdminPasswordKey: HARBOR_ADMIN_PASSWORD
harborAdminPassword: "<Password>"

caSecretName: ""

secretKey: "not-a-secure-key"
existingSecretSecretKey: ""

proxy:
  httpProxy:
  httpsProxy:
  noProxy: 127.0.0.1,localhost,.local,.internal
  components:
    - core
    - jobservice
    - trivy

enableMigrateHelmHook: false



nginx:
  image:
    repository: goharbor/nginx-photon
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:

portal:
  image:
    repository: goharbor/harbor-portal
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  serviceAnnotations: {}
  priorityClassName:

core:
  image:
    repository: goharbor/harbor-core
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  startupProbe:
    enabled: true
    initialDelaySeconds: 10and the storage is longhorn.
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  serviceAnnotations: {}
  configureUserSettings:
  quotaUpdateProvider: db # Or redis
  secret: ""
  existingSecret: ""
  secretName: ""
  tokenKey: |
  tokenCert: |
  xsrfKey: ""
  existingXsrfSecret: ""
  existingXsrfSecretKey: CSRF_KEY
  priorityClassName:
  artifactPullAsyncFlushDuration:
  gdpr:
    deleteUser: false
    auditLogsCompliant: false

jobservice:
  image:
    repository: goharbor/harbor-jobservice
    tag: v2.10.1and the storage is longhorn.
  replicas: 1
  revisionHistoryLimit: 10
  serviceAccountName: ""
  automountServiceAccountToken: false
  maxJobWorkers: 10
  jobLoggers:
    - file
  loggerSweeperDuration: 14 #days
  notification:
    webhook_job_max_retry: 3
    webhook_job_http_client_timeout: 3 # in seconds
  reaper:
    max_update_hours: 24
    max_dangling_hours: 168

  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints:
  podAnnotations: {}
  podLabels: {}
  secret: ""
  existingSecret: ""
  existingSecretKey: JOBSERVICE_SECRET
  priorityClassName:

registry:
  serviceAccountName: ""
  automountServiceAccountToken: false
  registry:
    image:
      repository: goharbor/registry-photon
      tag: v2.10.1
    extraEnvVars: []
  controller:
    image:
      repository: goharbor/harbor-registryctl
      tag: v2.10.1

    extraEnvVars: []
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:
  secret: ""
  existingSecret: ""
  existingSecretKey: REGISTRY_HTTP_SECRET
  relativeurls: false
  credentials:
    username: "harbor_registry_user"
    password: "harbor_registry_password"
    existingSecret: ""
    htpasswdString: ""
  middleware:
    enabled: false
    type: cloudFront
    cloudFront:
      baseurl: example.cloudfront.net
      keypairid: KEYPAIRID
      duration: 3000s
      ipfilteredby: none
      privateKeySecret: "my-secret"
  upload_purging:
    enabled: true
    age: 168h
    interval: 24h
    dryrun: false

trivy:
  enabled: true
  image:
    repository: goharbor/trivy-adapter-photon
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  debugMode: false
  vulnType: "os,library"
  severity: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
  ignoreUnfixed: false
  insecure: false
  gitHubToken: ""
  skipUpdate: false
  skipJavaDBUpdate: false  
  offlineScan: false
  securityCheck: "vuln"
  timeout: 5m0s
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1
      memory: 1Gi
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:

database:
  type: internal
  internal:
    serviceAccountName: ""
    automountServiceAccountToken: false
    image:and the storage is longhorn.
      repository: goharbor/harbor-db
      tag: v2.10.1
    password: "changeit"
    shmSizeLimit: 512Mi
    livenessProbe:
      timeoutSeconds: 1
    readinessProbe:
      timeoutSeconds: 1
    extraEnvVars: []
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName:
    initContainer:
      migrator: {}
      permissions: {}
  external:
    host: "192.168.0.1"
    port: "5432"
    username: "user"
    password: "password"
    coreDatabase: "registry"
    existingSecret: ""
    sslmode: "disable"
  maxIdleConns: 100
  maxOpenConns: 900
  podAnnotations: {}
  podLabels: {}

redis:
  type: internal
  internal:
    serviceAccountName: ""
    automountServiceAccountToken: false
    image:
      repository: goharbor/redis-photon
      tag: v2.10.1
    extraEnvVars: []
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName:
    jobserviceDatabaseIndex: "1"
    registryDatabaseIndex: "2"
    trivyAdapterIndex: "5"
  external:
    addr: "192.168.0.2:6379"
    sentinelMasterSet: ""
    coreDatabaseIndex: "0"
    jobserviceDatabaseIndex: "1"
    registryDatabaseIndex: "2"
    trivyAdapterIndex: "5"
    username: ""
    password: ""
    existingSecret: ""
  podAnnotations: {}
  podLabels: {}

exporter:
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  podAnnotations: {}
  podLabels: {}
  serviceAccountName: ""
  automountServiceAccountToken: false
  image:
    repository: goharbor/harbor-exporter
    tag: v2.10.1
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  cacheDuration: 23
  cacheCleanInterval: 14400
  priorityClassName:

metrics:
  enabled: false
  core:
    path: /metrics
    port: 8001
  registry:
    path: /metrics
    port: 8001
  jobservice:
    path: /metrics
    port: 8001
  exporter:
    path: /metrics
    port: 8001
  serviceMonitor:
    enabled: false
    additionalLabels: {}
    interval: ""
    metricRelabelings:
      []
    relabelings:
      []

trace:
  enabled: false
  provider: jaeger
  sample_rate: 1
  jaeger:
    endpoint: http://hostname:14268/api/traces
  otel:
    endpoint: hostname:4318
    url_path: /v1/traces
    compression: false
    insecure: true
    timeout: 10

cache:
  enabled: false
  expireHours: 24

Some of the logs while trying to push to the registry. Results in a 401. The docker login also results in 401.

[15/Apr/2024:10:58:59 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.002 0.002 .
[15/Apr/2024:10:58:59 +0000]:10.42.3.0 - "GET /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 200 633 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.009 0.009 .
[15/Apr/2024:10:59:00 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 200 2 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.068 0.069 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.002 0.002 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "GET /service/token?account=admin&scope=repository%3Atest-speed%2F1gb-random-file%3Apush%2Cpull&service=harbor-registry HTTP/1.1" 200 722 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.011 0.011 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:abf85a4cf2dd657ef8721648a7f6122c6758b9390220bc95e8369a9d1df123b4 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.003 0.003 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:b2388ca7fa65a68824f137dc4184ea3ea789570753d795042d9af40fc9383448 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.004 0.005 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:026cb769511a65d93a9a24aadff9124af782f85b558d1e981931d1f947ae2ee0 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.003 0.003 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.087 0.087 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.086 0.086 .
[15/Apr/2024:10:59:07 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.004 0.004 .
[15/Apr/2024:10:59:07 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.086 0.087 
image

I think the problem here is that the harbor as TLS Disabled and trys to connect to port 80 instead of port 443 and docker doenst allow that while trying to push a image (?).

Any help provided is welcome. Thanks.
@gsaudade99 gsaudade99 changed the title Harbor Docker Login 401 behin OPNSense Harbor Docker Login 401 behind OPNSense Apr 16, 2024
@zyyw zyyw added the help wanted Extra attention is needed label Apr 18, 2024
@zyyw zyyw self-assigned this Apr 18, 2024
@zyyw
Copy link
Collaborator

zyyw commented Apr 18, 2024

could you please try to config registry.relativeurls to be true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wy65701436 wy65701436

@zyyw zyyw

Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wy65701436 @zyyw @gsaudade99