ERROR: column t0.subject_artifact_repo does not exist at character 61

[reysravanga]

Hi Team,

I am unable to push docker images to Harbor and i get 500 internal server error. I have checked the harbor components i see few errors in harbor-core and postgress db . Looks like harbor core and db communication is broken.

prior to this issue, i have done the resizing of the harbor registry due to no space left on the device.

ERROR:

The push refers to repository [harbor.****.30002/project/image-service] 49aef3306d50: Layer already exists 739ad2bd66c0: Layer already exists 083d7f8cdb84: Layer already exists bc08f0a5078f: Layer already exists 9aef73d9cee1: Layer already exists b604c14a93b7: Layer already exists 8f80e4001bb6: Layer already exists 2e62ab2e522b: Layer already exists 3fc495a70927: Layer already exists ddab61a14989: Layer already exists 6e7022fb2e3d: Layer already exists 004040371961: Layer already exists b2e5b1eee192: Layer already exists b485c6cd33a6: Layer already exists 6aa872026017: Layer already exists 43ba18a5eaf8: Layer already exists ff61a9b258e5: Layer already exists received unexpected HTTP status: 500 Internal Server Error

Harbor-core:

2024-04-09T11:26:48Z [INFO] [/controller/registry/controller.go:222]: Start regular health check for registries with interval 5m0s
2024-04-09T11:27:29Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="09d6e671-54ae-4b44-9632-21435e7629f5"]: failed to list accessory artifact: sha256:0848be9906a77319c727c28e007f851581ec04b5071ee2cd47b7b1d3c97d17e5, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:27:29Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:27:29Z [WARNING] [/controller/quota/controller.go:402][requestID="09d6e671-54ae-4b44-9632-21435e7629f5"]: rollback resources {"storage":3895} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-04-09T11:32:16Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="b6fdd65c-0ae9-4c06-8002-bb72d86f32ca"]: failed to list accessory artifact: sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:32:16Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:32:16Z [WARNING] [/controller/quota/controller.go:402][requestID="b6fdd65c-0ae9-4c06-8002-bb72d86f32ca"]: rollback resources {"storage":527} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-04-09T11:36:08Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="c4512bd7-eda4-49d2-9ce3-8dd159fd2408"]: failed to list accessory artifact: sha256:0848be9906a77319c727c28e007f851581ec04b5071ee2cd47b7b1d3c97d17e5, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:36:08Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:36:08Z [WARNING] [/controller/quota/controller.go:402][requestID="c4512bd7-eda4-49d2-9ce3-8dd159fd2408"]: rollback resources {"storage":3895} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-04-09T11:36:34Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="4410aa0b-2adf-4981-bf64-30325f01190b"]: failed to list accessory artifact: sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:36:34Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:36:34Z [WARNING] [/controller/quota/controller.go:402][requestID="4410aa0b-2adf-4981-bf64-30325f01190b"]: rollback resources {"storage":527} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-04-09T11:37:14Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column s.critical_cnt does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:39:00Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="294477da-9072-4fd1-a68f-da79cf3c9622"]: failed to list accessory artifact: sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:39:00Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:39:00Z [WARNING] [/controller/quota/controller.go:402][requestID="294477da-9072-4fd1-a68f-da79cf3c9622"]: rollback resources {"storage":527} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-04-09T11:39:28Z [ERROR] [/server/middleware/subject/subject.go:161][middleware="subject" requestID="7b86f960-e999-41f9-8d4d-a45c92f56dcd"]: failed to list accessory artifact: sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335, error: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)
2024-04-09T11:39:28Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: column t0.subject_artifact_repo does not exist (SQLSTATE 42703)"}]}
2024-04-09T11:39:28Z [WARNING] [/controller/quota/controller.go:402][requestID="7b86f960-e999-41f9-8d4d-a45c92f56dcd"]: rollback resources {"storage":527} for project 2 failed, error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)

harbor-db:

2024-04-09 11:23:34.663 UTC [9] LOG:  redo done at 0/10D06DC0 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2024-04-09 11:23:34.692 UTC [7] LOG:  checkpoint starting: end-of-recovery immediate wait
2024-04-09 11:23:34.727 UTC [7] LOG:  checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.009 s, sync=0.007 s, total=0.042 s; sync files=3, longest=0.003 s, average=0.003 s; distance=0 kB, estimate=0 kB
2024-04-09 11:23:34.737 UTC [1] LOG:  database system is ready to accept connections
2024-04-09 11:27:29.615 UTC [283] ERROR:  column t0.subject_artifact_repo does not exist at character 61
2024-04-09 11:27:29.615 UTC [283] STATEMENT:  SELECT T0."id", T0."artifact_id", T0."subject_artifact_id", T0."subject_artifact_repo", T0."subject_artifact_digest", T0."type", T0."size", T0."digest", T0."creation_time" FROM "artifact_accessory" T0 WHERE T0."subject_artifact_digest" = $1 AND T0."subject_artifact_repo" = $2
2024-04-09 11:27:29.615 UTC [283] ERROR:  current transaction is aborted, commands ignored until end of transaction block
2024-04-09 11:27:29.615 UTC [283] STATEMENT:  SELECT "id", "reference", "reference_id", "hard", "creation_time", "update_time", "version" FROM "quota" WHERE "reference" = $1 AND "reference_id" = $2
2024-04-09 11:28:34.827 UTC [7] LOG:  checkpoint starting: time
2024-04-09 11:28:41.494 UTC [7] LOG:  checkpoint complete: wrote 68 buffers (0.4%); 0 WAL file(s) added, 0 removed, 0 recycled; write=6.639 s, sync=0.007 s, total=6.668 s; sync files=42, longest=0.006 s, average=0.001 s; distance=225 kB, estimate=225 kB
2024-04-09 11:32:16.631 UTC [283] ERROR:  column t0.subject_artifact_repo does not exist at character 61
2024-04-09 11:32:16.631 UTC [283] STATEMENT:  SELECT T0."id", T0."artifact_id", T0."subject_artifact_id", T0."subject_artifact_repo", T0."subject_artifact_digest", T0."type", T0."size", T0."digest", T0."creation_time" FROM "artifact_accessory" T0 WHERE T0."subject_artifact_digest" = $1 AND T0."subject_artifact_repo" = $2
2024-04-09 11:32:16.631 UTC [283] ERROR:  current transaction is aborted, commands ignored until end of transaction block
2024-04-09 11:32:16.631 UTC [283] STATEMENT:  SELECT "id", "reference", "reference_id", "hard", "creation_time", "update_time", "version" FROM "quota" WHERE "reference" = $1 AND "reference_id" = $2
2024-04-09 11:33:34.579 UTC [7] LOG:  checkpoint starting: time
2024-04-09 11:33:38.652 UTC [7] LOG:  checkpoint complete: wrote 42 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.035 s, sync=0.012 s, total=4.073 s; sync files=35, longest=0.010 s, average=0.001 s; distance=129 kB, estimate=215 kB
2024-04-09 11:36:08.988 UTC [849] ERROR:  column t0.subject_artifact_repo does not exist at character 61
2024-04-09 11:36:08.988 UTC [849] STATEMENT:  SELECT T0."id", T0."artifact_id", T0."subject_artifact_id", T0."subject_artifact_repo", T0."subject_artifact_digest", T0."type", T0."size", T0."digest", T0."creation_time" FROM "artifact_accessory" T0 WHERE T0."subject_artifact_digest" = $1 AND T0."subject_artifact_repo" = $2
2024-04-09 11:36:08.988 UTC [849] ERROR:  current transaction is aborted, commands ignored until end of transaction block
2024-04-09 11:36:08.988 UTC [849] STATEMENT:  SELECT "id", "reference", "reference_id", "hard", "creation_time", "update_time", "version" FROM "quota" WHERE "reference" = $1 AND "reference_id" = $2
2024-04-09 11:36:34.245 UTC [846] ERROR:  column t0.subject_artifact_repo does not exist at character 61
2024-04-09 11:36:34.245 UTC [846] STATEMENT:  SELECT T0."id", T0."artifact_id", T0."subject_artifact_id", T0."subject_artifact_repo", T0."subject_artifact_digest", T0."type", T0."size", T0."digest", T0."creation_time" FROM "artifact_accessory" T0 WHERE T0."subject_artifact_digest" = $1 AND T0."subject_artifact_repo" = $2
2024-04-09 11:36:34.245 UTC [846] ERROR:  current transaction is aborted, commands ignored until end of transaction block
2024-04-09 11:36:34.245 UTC [846] STATEMENT:  SELECT "id", "reference", "reference_id", "hard", "creation_time", "update_time", "version" FROM "quota" WHERE "reference" = $1 AND "reference_id" = $2
2024-04-09 11:37:14.186 UTC [1121] ERROR:  column s.critical_cnt does not exist at character 12
2024-04-09 11:37:14.186 UTC [1121] STATEMENT:  select sum(s.critical_cnt) critical_cnt,
               sum(s.high_cnt)     high_cnt,
               sum(s.medium_cnt)   medium_cnt,
               sum(s.low_cnt)      low_cnt,
               sum(s.none_cnt)     none_cnt,
               sum(s.unknown_cnt)  unknown_cnt,
               sum(s.fixable_cnt)  fixable_cnt
        from artifact a
                 left join scan_report s on a.digest = s.digest
                 where s.registration_uuid = $1
2024-04-09 11:38:34.745 UTC [7] LOG:  checkpoint starting: time
2024-04-09 11:38:40.407 UTC [7] LOG:  checkpoint complete: wrote 58 buffers (0.4%); 0 WAL file(s) added, 0 removed, 0 recycled; write=5.631 s, sync=0.009 s, total=5.663 s; sync files=35, longest=0.008 s, average=0.001 s; distance=229 kB, estimate=229 kB
2024-04-09 11:39:00.497 UTC [1524] ERROR:  column t0.subject_artifact_repo does not exist at character 61
2024-04-09 11:39:00.497 UTC [1524] STATEMENT:  SELECT T0."id", T0."artifact_id", T0."subject_artifact_id", T0."subject_artifact_repo", T0."subject_artifact_digest", T0."type", T0."size", T0."digest", T0."creation_time" FROM "artifact_accessory" T0 WHERE T0."subject_artifact_digest" = $1 AND T0."subject_artifact_repo" = $2
```

**harbor core version **
VERSION_TAG: v2.11.0-build.2485
REGISTRY_VERSION: v2.8.3-patch-redis
TRIVY_VERSION: v0.47.0
TRIVY_ADAPTER_VERSION: v0.30.19 

[MinerYang]

Is it a fresh install harbor or upgrade from specific version? Any db dump or alternation during upgrade? Please provide more about the process.

[reysravanga]

It was only Registry Disk size resize , and not a version upgrade,

 sudo helm history harbor -n harbor
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
1               Wed Feb 22 22:41:19 2023        superseded      harbor-1.11.0   2.7.0           Install complete
2               Tue Mar 28 19:37:55 2023        superseded      harbor-1.11.0   2.7.0           Upgrade complete
3               Fri Mar 31 09:14:27 2023        superseded      harbor-1.11.0   2.7.0           Upgrade complete
4               Sun Apr  7 07:58:01 2024        deployed        harbor-1.11.0   2.7.0           Upgrade complete

there was no changes to DB/Harbor-core

[reysravanga]

Can Someone check this issue ?

[zyyw]

It was only Registry Disk size resize , and not a version upgrade,

 sudo helm history harbor -n harbor
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
1               Wed Feb 22 22:41:19 2023        superseded      harbor-1.11.0   2.7.0           Install complete
2               Tue Mar 28 19:37:55 2023        superseded      harbor-1.11.0   2.7.0           Upgrade complete
3               Fri Mar 31 09:14:27 2023        superseded      harbor-1.11.0   2.7.0           Upgrade complete
4               Sun Apr  7 07:58:01 2024        deployed        harbor-1.11.0   2.7.0           Upgrade complete

there was no changes to DB/Harbor-core

Based on the helm history output and the this info from the description of this issue:

**harbor core version **
VERSION_TAG: v2.11.0-build.2485

Harbor recommends to upgrade within every n-2 version if you are trying to upgrade Harbor version.

[zyyw]

btw, according to this info:

**harbor core version **
VERSION_TAG: v2.11.0-build.2485

it seems that you are trying to install with / upgrade to the build of Harbor main branch. Please use a release version, instead of using main branch.