Garbage collector not clean redis keys

[nevlkv]

Expected behavior and actual behavior:
Garbage collector not cleanup redis keys, and failed with message

2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 2937 blobs and 812 manifests are actually deleted 2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 54343 MB space. 2023-09-06T02:37:23Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

Steps to reproduce the problem:

Starts registry cleanup

Versions:

• harbor version: 2.8.4

Additional context:

external redis with default user

_REDIS_URL_REG: redis://:password@harbor-redis-master:6379/2?idle_timeout_seconds=30

[wy65701436]

the error points the auth info is incorrect, can you check that at your end?

[nevlkv]

password correct

redis-cli -n 2 -a <password>
127.0.0.1:6379[2]> info keyspace
# Keyspace
db0:keys=133200,expires=133085,avg_ttl=2212096
db1:keys=2036,expires=1748,avg_ttl=83530815
db2:keys=5784,expires=156,avg_ttl=85886397
db5:keys=63,expires=54,avg_ttl=2360250

[nevlkv]

garbage service used _REDIS_URL_REG: ?

[MinerYang]

Hi @nevlkv

• Is this a fresh install harbor? Have you changed the redis password after installation?
• Could you also check/provide other redis url, for example _REDIS_URL_CORE?

[nevlkv]

1. current instance was upgraded from 2.7.1, redis is external and not changed
2. we use default redis account and password always identical

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

i can auth with via redis-cli and flushdb

GC use same connection string

[viceice]

Seeing same error:

023-09-17T00:02:59Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:515]: failed to clean registry cache failed to scan keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

I've no password for my external redis, i'm using network policies to limit redis access to harbor.

_REDIS_URL_CORE: redis://keydb:6379/0?idle_timeout_seconds=30
_REDIS_URL_REG: redis://keydb:6379/2?idle_timeout_seconds=30

Harbor version: v2.9.0

[viceice]

It seems the redis url is not passed to the job service via env.

I only see this inside config.yml if job service:

worker_pool:
  workers: 10
  backend: "redis"
  redis_pool:
    redis_url: "redis://keydb:6379/1"
    namespace: "harbor_job_service_namespace"
    idle_timeout_second: 3600

[viceice]

We probably need to set ?

_REDIS_URL_REG: "{{ template "harbor.redis.urlForRegistry" . }}"

harbor-helm/templates/jobservice/jobservice-cm-env.yaml

Line 15 in 86dd759

JOBSERVICE_WEBHOOK_JOB_HTTP_CLIENT_TIMEOUT: "{{ .Values.jobservice.notification.webhook_job_http_client_timeout }}"

[nevlkv]

error stable

2023-09-21T07:55:57Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'GARBAGE_COLLECTION:c09d285cf0794f361744acab' exit with error: run error: error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled.

[nevlkv]

Current state

2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 3177 blobs and 562 manifests are actually deleted
2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 37506 MB space.
2023-09-24T02:31:20Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

[MinerYang]

Hi @nevlkv ,
Could you try to monitor the registry log when pushing images, if there's any error msg?

[nevlkv]

Sometimes on pull 404
with "blob unknown"

time="2023-09-26T11:19:16.21575386Z" level=error msg="response completed with error" auth.user.name=harbor err.code="blob unknown" err.detail=sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86 err.message="blob unknown to registry" go.version=go1.20.7 http.request.host=REGISTRY http.request.id=cdc0816e-2548-40d7-b33c-3f2f9a26837b http.request.method=HEAD http.request.remoteaddr=*.*.*.* http.request.uri="/v2/REPO/blobs/sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" http.request.useragent="docker/20.10.22 go/go1.18.9 git-commit/42c8b31 kernel/5.15.0-83-generic os/linux arch/amd64 UpstreamClient(docker-compose/1.29.2 docker-py/6.1.3 Linux/5.15.0-83-generic)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.591881ms http.response.status=404 http.response.written=157 vars.digest="sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" vars.name="REPO"

[nevlkv]

If default user in URI undefined rise error

$ redis-cli -u redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
$ redis-cli -u redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Keyspace
db0:keys=1732,expires=1732,avg_ttl=1826800
db1:keys=116,expires=65,avg_ttl=44559991
db2:keys=1,expires=0,avg_ttl=0
db5:keys=4,expires=0,avg_ttl=0

define user in values

   type: external
   external:
     addr: "harbor-redis-master:6379"
+    username: "default"

and

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

became

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://default:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

and GC success !!!

[viceice]

Interestingly a manual run works, but the sheduled runs are always failing. 😕

[nevlkv]

same error after upgrade to 2.10

scheduled fail, manual success

upgraded 08/02/2024

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[viceice]

not stale 😞

[nevlkv]

successful only manual job

[nevlkv]

execute success only manual