Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload Aliyun OSS issue #877

Open
dxygit1 opened this issue Mar 26, 2024 · 11 comments
Open

Upload Aliyun OSS issue #877

dxygit1 opened this issue Mar 26, 2024 · 11 comments
Assignees
@Slach
Milestone
2.6.0

Comments

@dxygit1
Copy link

dxygit1 commented Mar 26, 2024

config

      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup
      - name: S3_ACCESS_KEY
        value: ak
      - name: S3_SECRET_KEY
        value: ak
      - name: S3_FORCE_PATH_STYLE
        value: "true"
      - name: S3_DISABLE_SSL
        value: "true"
      - name: S3_DEBUG
        value: "true

error info

 info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6602A41480647F39337A1EAC, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.

image

Could you please help me see where I filled in wrong?
@Slach
Copy link
Collaborator

Slach commented Mar 26, 2024

Try to remove

      - name: S3_DISABLE_SSL
        value: "true"

and according to
#505 (comment)
and your shared error message
try to use following URL

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_FORCE_PATH_STYLE
        value: "false"

@Slach Slach closed this as completed Mar 26, 2024
@dxygit1
Copy link
Author

dxygit1 commented Mar 26, 2024

Following your instructions will result in the following error.

2024/03/26 13:27:51.345553  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Tue, 26 Mar 2024 13:27:37 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6602CD496F20953034A2F02E
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 2


2024/03/26 13:27:51.345754  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6602CD496F20953034A2F02E, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/26 13:27:51.345815  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 6602CD496F20953034A2F02E, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/26 13:27:51.346549 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/26 13:27:51.346659  info clickhouse connection closed logger=clickhouse

@Slach
Copy link
Collaborator

Slach commented Mar 26, 2024

could you provide

[s3:DEBUG] Request which happends before response?

@dxygit1
Copy link
Author

dxygit1 commented Mar 27, 2024 

2024/03/27 01:51:10.263948  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.266304  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.266397  info Create integration tables logger=server
2024/03/27 01:51:10.266420  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.267343  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.267427  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.270410  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.274431  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.279339  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 01:51:10.281289  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 01:51:10.282350  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 01:51:10.287803  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 01:51:10.291312  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 01:51:10.292185  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 01:51:10.297966  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.298356  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 01:51:10.300054  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 01:51:10.300165  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.300302  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.300956  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.300977  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.301206  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.301242  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.302533  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.302770  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.305608  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.306309  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.309160 error ResumeOperationsAfterRestart return error: open /var/lib/clickhouse/backup: no such file or directory logger=server.Run
2024/03/27 01:51:10.309786  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.309822  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.311819  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.311865  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 01:51:10.314505  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 01:51:10.315640  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 01:51:10.318045  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 01:51:10.322159  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 32533705-7dfd-467b-b395-9564b69a34d5
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=aa717b3683089d17111e98a923659b200d267b12776409dd77a4384e18e021cd
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T015110Z


2024/03/27 01:51:10.718922  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 01:50:56 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 66037B80C505B33730A9B573
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 122


2024/03/27 01:51:10.719793  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 66037B80C505B33730A9B573, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 01:51:10.719950 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 01:51:10.721128  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 44d0e9db-684f-45f3-bcdd-e44baf4dd3c0
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T015110Z


2024/03/27 01:51:10.734252  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 01:50:56 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 66037B80C505B33730A6B773
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 4


2024/03/27 01:51:10.734472  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 66037B80C505B33730A6B773, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 01:51:10.734533  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 66037B80C505B33730A6B773, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/27 01:51:10.735120 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 01:51:10.735264  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.735299  info Update backup metrics finish LastBackupCreateLocal=<nil> LastBackupCreateRemote=<nil> LastBackupSizeLocal=0 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=0 NumberBackupsRemote=0 duration=435ms logger=server

@Slach
Copy link
Collaborator

Slach commented Mar 27, 2024

GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com

Bucket in path and custom endpoint which already contains bucket name

@dxygit1
are you sure you applied
force_path_style: false?

Let's remove bucket name from endpoint and could you share logs for the following approach?

      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_FORCE_PATH_STYLE
        value: "true"

@Slach Slach mentioned this issue Mar 27, 2024
Closed
@dxygit1
Copy link
Author

dxygit1 commented Mar 27, 2024

S3_FORCE_PATH_STYLE has been set to false.

    env:
      - name: CLICKHOUSE_PASSWORD
        value: abc123
      - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
        value: "false"
      - name: LOG_LEVEL
        value: debug
      - name: ALLOW_EMPTY_BACKUPS
        value: "true"
      - name: API_LISTEN
        value: 0.0.0.0:7171
      - name: API_CREATE_INTEGRATION_TABLES
        value: "true"
      - name: BACKUPS_TO_KEEP_REMOTE
        value: "3"
      - name: REMOTE_STORAGE
        value: s3
      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_ACCESS_KEY
        value: xxxx
      - name: S3_SECRET_KEY
        value: xxxx
      - name: S3_FORCE_PATH_STYLE
        value: "false"
      - name: S3_DEBUG
        value: "true"

error info

2024/03/27 05:08:27.600610  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:27.601825 error clickhouse connection ping: tcp://localhost:9000 return error: dial tcp [::1]:9000: connect: connection refused logger=clickhouse
2024/03/27 05:08:27.601848 error dial tcp [::1]:9000: connect: connection refused logger=server.Run
2024/03/27 05:08:32.603032  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.604575  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.604695  info Create integration tables logger=server
2024/03/27 05:08:32.604728  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.606200  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.606314  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.608917  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.612406  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.616601  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:08:32.618099  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 05:08:32.619185  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:08:32.775490  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:08:32.777477  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 05:08:32.778511  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:08:32.785289  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.785770  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 05:08:32.787630  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 05:08:32.787655  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.787723  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.789457  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.789483  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.789561  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.789592  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.791280  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.791290  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.794952  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.795223  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.801080  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.801133  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.801852  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.801901  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:08:32.804261  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:08:32.805496  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:08:32.807831  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:08:32.812084  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: f3ec91f5-af08-4f70-8890-eea9733e9976
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T050832Z


2024/03/27 05:08:32.914038  info [s3:DEBUG] Response
HTTP/1.1 403 Forbidden
Content-Length: 368
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:08:18 GMT
Server: AliyunOSS
X-Amz-Request-Id: 6603A9C22B654B303133FC64
X-Oss-Ec: 0003-00001401


2024/03/27 05:08:32.914850  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6603A9C22B654B303133FC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.
2024/03/27 05:08:32.914959 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 05:08:32.915631  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 4ba83a92-a939-4fa1-9015-425910836b6b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=759e7bbee2ebfbba9c07886ddd968570afc75cbd3c263c6039693d86cb04d3ef
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T050832Z


2024/03/27 05:08:32.927442  info [s3:DEBUG] Response
HTTP/1.1 403 Forbidden
Content-Length: 368
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:08:18 GMT
Server: AliyunOSS
X-Amz-Request-Id: 6603A9C22B654B3031CDFC64
X-Oss-Ec: 0003-00001401


2024/03/27 05:08:32.927621  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6603A9C22B654B3031CDFC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.
2024/03/27 05:08:32.927669  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 6603A9C22B654B3031CDFC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access. logger=s3
2024/03/27 05:08:32.927959 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 05:08:32.928052  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.928082  info Update backup metrics finish LastBackupCreateLocal=2024-03-27 03:38:18.515102036 +0000 UTC LastBackupCreateRemote=<nil> LastBackupSizeLocal=687823 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=2 NumberBackupsRemote=0 duration=141ms logger=server

@Slach
Copy link
Collaborator

Slach commented Mar 27, 2024 •
edited

let's apply

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_FORCE_PATH_STYLE
        value: "false"

and please share the logs

@Slach Slach reopened this Mar 27, 2024
@Slach Slach added this to the 2.6.0 milestone Mar 27, 2024
@Slach Slach self-assigned this Mar 27, 2024
@dxygit1
Copy link
Author

dxygit1 commented Mar 27, 2024 

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_ACCESS_KEY
        value: xxx
      - name: S3_SECRET_KEY
        value: xxx
      - name: S3_FORCE_PATH_STYLE
        value: "false"

2024/03/27 05:39:16.312745  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:16.315045  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:16.315134  info Create integration tables logger=server
2024/03/27 05:39:16.315162  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:16.316044  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:16.316135  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:16.319549  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:16.323973  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:16.329320  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:39:16.330692  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 05:39:16.331772  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:39:17.024788  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:39:17.026997  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 05:39:17.027937  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:39:17.032606  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.033108  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 05:39:17.035021  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 05:39:17.035088  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.035109  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.036207  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.036240  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:17.036438  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.036460  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:17.038041  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:17.038045  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:17.041939  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:17.042978  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:17.049299  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.049350  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.050197  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.050251  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:39:17.052450  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:39:17.053496  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:39:17.055560  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:39:17.059445  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9b075f36-e76e-4bff-b498-f34b5f2767ad
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=d9a40ad221e7c608de65011088d3af0ffd2a31c91b216eb03832c6e11be9a246
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T053917Z


2024/03/27 05:39:17.221982  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:39:02 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6603B0F6AF96A13139D7F1F7
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 66


2024/03/27 05:39:17.222652  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139D7F1F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 05:39:17.222769 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 05:39:17.223542  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: eff8cdd0-6f43-4ac2-b9bd-46680cc97f8c
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T053917Z


2024/03/27 05:39:17.238089  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:39:02 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6603B0F6AF96A13139E0F2F7
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 6


2024/03/27 05:39:17.238263  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139E0F2F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 05:39:17.238317  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139E0F2F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/27 05:39:17.238620 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 05:39:17.238727  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.238758  info Update backup metrics finish LastBackupCreateLocal=2024-03-27 03:38:18.515102036 +0000 UTC LastBackupCreateRemote=<nil> LastBackupSizeLocal=687823 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=2 NumberBackupsRemote=0 duration=204ms logger=server

@Slach
Copy link
Collaborator

Slach commented Mar 27, 2024

ok. it looks like some bug in AWS SDK or iside our confugation of AWS SDK

@dxygit1
Copy link
Author

dxygit1 commented Mar 27, 2024

Okay, let's use MinIO as a replacement first. I've tested MinIO, and it's working fine. Could you please kindly help fix this bug? We really need this functionality

@dxygit1
Copy link
Author

dxygit1 commented Apr 2, 2024 •
edited

May I ask when it will support Alibaba Cloud OSS?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Slach Slach

Labels
None yet
Projects
None yet
Milestone
2.6.0
Development

No branches or pull requests
2 participants
@Slach @dxygit1