error can't copy data to detached

[bmailhe]

When trying to restore a backup on another cluster I get this error:

2023/12/15 14:36:09.767227 error can't copy data to datached 'mydb.my_table': 
dstDataPaths=map[string]string{
"ssd_sdb":"/var/lib/private/clickhouse/sdb/store/7e8/7e8fb205-fa31-4fb2-9a76-e4d3aaa3a423/", 
"ssd_sdc":"/var/lib/private/clickhouse/sdc/store/7e8/7e8fb205-fa31-4fb2-9a76-e4d3aaa3a423/", 
"ssd_sdd":"/var/lib/private/clickhouse/sdd/store/7e8/7e8fb205-fa31-4fb2-9a76-e4d3aaa3a423/", 
"ssd_sde":"/var/lib/private/clickhouse/sde/store/7e8/7e8fb205-fa31-4fb2-9a76-e4d3aaa3a423/"
}, 
not contains default

The destination cluster is different than the source cluster.

The dst cluster has jbod configured:

mycluster :) SELECT * FROM system.storage_policies
┌─policy_name─┬─volume_name─┬─volume_priority─┬─disks─────────────────────────────────────┬─volume_type─┬─max_data_part_size─┬─move_factor─┬─prefer_not_to_merge─┬─perform_ttl_move_on_insert─┬─load_balancing─┐
│ default     │ default     │               1 │ ['ssd_sdb','ssd_sdc','ssd_sdd','ssd_sde'] │ JBOD        │                  0 │           0 │                   0 │                          1 │ ROUND_ROBIN    │
└─────────────┴─────────────┴─────────────────┴───────────────────────────────────────────┴─────────────┴────────────────────┴─────────────┴─────────────────────┴────────────────────────────┴────────────────┘

mycluster :) SELECT * FROM system.disks
┌─name────┬─path─────────────────────────────┬─────free_space─┬────total_space─┬─unreserved_space─┬─keep_free_space─┬─type──┬─is_encrypted─┬─is_read_only─┬─is_write_once─┬─is_remote─┬─is_broken─┬─cache_path─┐
│ default │ /var/lib/clickhouse/             │   433123758080 │   470688030720 │     433123758080 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
│ ssd_sdb │ /var/lib/private/clickhouse/sdb/ │ 13888994062336 │ 13889010892800 │   13888994062336 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
│ ssd_sdc │ /var/lib/private/clickhouse/sdc/ │ 13888994066432 │ 13889010892800 │   13888994066432 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
│ ssd_sdd │ /var/lib/private/clickhouse/sdd/ │ 13888994066432 │ 13889010892800 │   13888994066432 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
│ ssd_sde │ /var/lib/private/clickhouse/sde/ │ 13888994066432 │ 13889010892800 │   13888994066432 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
└─────────┴──────────────────────────────────┴────────────────┴────────────────┴──────────────────┴─────────────────┴───────┴──────────────┴──────────────┴───────────────┴───────────┴───────────┴────────────┘

while the src cluster has :

mycluster :) SELECT * FROM system.storage_policies
┌─policy_name─┬─volume_name─┬─volume_priority─┬─disks───────┬─volume_type─┬─max_data_part_size─┬─move_factor─┬─prefer_not_to_merge─┬─perform_ttl_move_on_insert─┬─load_balancing─┐
│ default     │ default     │               1 │ ['default'] │ JBOD        │                  0 │           0 │                   0 │                          1 │ ROUND_ROBIN    │
└─────────────┴─────────────┴─────────────────┴─────────────┴─────────────┴────────────────────┴─────────────┴─────────────────────┴────────────────────────────┴────────────────┘
mycluster :) SELECT * FROM system.disks
┌─name────┬─path─────────────────┬─────free_space─┬────total_space─┬─unreserved_space─┬─keep_free_space─┬─type──┬─is_encrypted─┬─is_read_only─┬─is_write_once─┬─is_remote─┬─is_broken─┬─cache_path─┐
│ default │ /var/lib/clickhouse/ │ 24169531301888 │ 41832927952896 │   24152346233100 │               0 │ local │            0 │            0 │             0 │         0 │         0 │            │
└─────────┴──────────────────────┴────────────────┴────────────────┴──────────────────┴─────────────────┴───────┴──────────────┴──────────────┴───────────────┴───────────┴───────────┴────────────┘

it seems related to the jbod configuration, how can I fix this?

[Slach]

looks like you move your backup with rsync and didn't use remote storage? am i right?

Duplicated with #561
plan to fix in 2.5.0 (this is not easy as expected)

try to apply recommendations from #561 (comment)

you have two options here:

1. add default disk to your default JBOD volume on destination with move_factor 0.1 for example and successfully restore backup if you have enough space in /var/lib/clickhouse/backup, after it apply ALTER TABLE db.table MOVE PART TO DISK 'ssd_XXX' to force move data between disks

2. manually move parts from /var/lib/clickhouse/backup/backup_name/shadow/db/table/default/<part_name> to /var/lib/private/clickhouse/sdX/backup/backup_name/shadow/db/table/sdX/<part_name>
   and manually change parts section in /var/lib/clickhouse/backup/backup_name/metadata/db/table.json

[bmailhe]

I use rclone to move backups to and from S3 BackBlaze.