Milvus backup tool: How to speed up restore of 200 million vectors? #32750

schuberttobias · 2024-04-30T11:03:06Z

schuberttobias
Apr 30, 2024

Hi!

I'm trying to restore 200 million vectors using Milvus backup tool v0.4.12 (https://github.com/zilliztech/milvus-backup/).

Restore and create index takes 6 hours. Ideal would be <1 hour.

Problem

./milvus-backup create takes 150 seconds.
./milvus-backup restore --name backup_2024_04_26_14_48_11_149630773 takes 4 hours. After that create index takes ca. 2 hours.
CPU utilization on the data node (green) is always <1% during restore.
Milvus Backup Tool process only uses 1 CPU core.
Increasing backup.parallelism.restoreCollection to 2 and 8 as suggested in https://stackoverflow.com/a/77878275 did not help. Milvus backup process still only uses 1 CPU core. CPU utilization on data node is still low. Speed does not improve.
Using 4 data nodes as suggested in https://stackoverflow.com/a/7787827 did not help. CPU utilization on data nodes is still below 1% during restore. Speed did not improve.
I use the same S3 bucket for bucketName and backupBucketName.

Questions

What can I do to make restore faster?
Milvus backup tool's log looks like it does bulk insert serially, not in parallel. Parallelising bulk insert to speed up restore was implemented in support concurrent bulkinsert zilliztech/milvus-backup#250. But the PR was never merged?
I could write a script that calls https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md one time and put all the files created by Milvus backup into the files argument. Is that supported? Will that use "Refactored BulkInsert" from Milvus v2.4.0 (https://milvus.io/docs/release_notes.md#Refactored-BulkInsert) and be faster than Milvus Backup tool?

Thanks!

Details

Cluster

Milvus v2.4.0 on AWS EKS with two c6i.4xlarge.
Backup tool runs on c7g.2xlarge (arm64) outside of the cluster.

show collection

milvus_cli > show collection -c fingerprints
+---------------+--------------------------------------+
| Name          | fingerprints                   |
+---------------+--------------------------------------+
| Description   | Default schema                       |
+---------------+--------------------------------------+
| Is Empty      | False                                |
+---------------+--------------------------------------+
| Entities      | 192209228                            |
+---------------+--------------------------------------+
| Primary Field | primary_id                           |
+---------------+--------------------------------------+
| Schema        | Description: Default schema          |
|               |                                      |
|               | Auto ID: True                        |
|               |                                      |
|               | Fields(* is the primary field):      |
|               |  - *primary_id INT64                 |
|               |  - t_id INT64                    |
|               |  - segment_id INT64                  |
|               |  - fingerprint FLOAT_VECTOR dim: 128 |
+---------------+--------------------------------------+
| Partitions    | - _default                           |
+---------------+--------------------------------------+
| Indexes       | - fingerprint                        |
+---------------+--------------------------------------+

Milvus backup tool config

ubuntu@ip-10-0-63-121:~/milvus-backup$ cat configs/backup.yaml
# Configures the system log output.
log:
  level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
  console: true # whether print log to console
  file:
    rootPath: "configs/backup.log"

http:
  simpleResponse: true

# milvus proxy address, compatible to milvus.yaml
milvus:
  address: redacted.eu-central-1.elb.amazonaws.com
  port: 19530
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  # tlsMode: 0
  # user: "root"
  # password: "Milvus"

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "s3" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)

  address: s3.eu-central-1.amazonaws.com # Address of MinIO/S3
  port: 80   # Port of MinIO/S3
  accessKeyID: redacted # accessKeyID of MinIO/S3
  secretAccessKey: redacted  # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""

  bucketName: "kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # only for azure
  # backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  # backupSecretAccessKey: minioadmin # MinIO/S3 encryption string

  backupBucketName: "kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

backup:
  maxSegmentGroupSize: 2G

  parallelism:
    # collection level parallelism to backup
    backupCollection: 4
    # thread pool to copy data. reduce it if blocks your storage's network bandwidth
    copydata: 128
    # Collection level parallelism to restore
    restoreCollection: 2

  # keep temporary files during restore, only use to debug
  keepTempFiles: false

  # Pause GC during backup through Milvus Http API.
  gcPause:
    enable: true
    seconds: 7200
    address: http://redacted.eu-central-1.elb.amazonaws.com:9091

Milvus backup log

[2024/04/29 20:13:17.478 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=configs/backup.log]
[2024/04/29 20:13:17.478 +00:00] [INFO] [cmd/restore.go:46] ["restore cmd input args"] [args="[]"]
[2024/04/29 20:13:17.478 +00:00] [INFO] [core/backup_impl_restore_backup.go:26] ["receive RestoreBackupRequest"] [requestId=eaa27700-0664-11ef-be12-02e155b13723] [backupName=backup_2024_04_28_18_12_36_634313588] [onlyMeta=false] [restoreIndex=false] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=fa
lse] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=]
[2024/04/29 20:13:17.478 +00:00] [INFO] [core/backup_context.go:106] ["{Base:0x40001cc500 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://redacted.eu-central-1.elb.amazonaws.com:9091}"]
[2024/04/29 20:13:17.478 +00:00] [INFO] [core/backup_context.go:107] ["{Base:0x40001cc500 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/04/29 20:13:17.478 +00:00] [INFO] [core/backup_context.go:209] ["receive GetBackupRequest"] [requestId=eaa27caf-0664-11ef-be12-02e155b13723] [backupName=backup_2024_04_28_18_12_36_634313588] [backupId=] [bucketName=] [path=]
[2024/04/29 20:13:17.506 +00:00] [INFO] [storage/minio_chunk_manager.go:151] ["minio chunk manager init success."] [bucketname=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [root=]
[2024/04/29 20:13:17.660 +00:00] [INFO] [core/backup_context.go:301] ["finish GetBackupRequest"] [requestId=eaa27caf-0664-11ef-be12-02e155b13723] [backupName=backup_2024_04_28_18_12_36_634313588] [backupId=] [bucketName=] [path=]
[2024/04/29 20:13:17.660 +00:00] [INFO] [core/backup_impl_restore_backup.go:178] ["Collections to restore"] [collection_num=1]
[2024/04/29 20:13:17.674 +00:00] [INFO] [core/backup_impl_restore_backup.go:331] ["Start collection level restore pool"] [parallelism=2]
[2024/04/29 20:13:17.674 +00:00] [INFO] [core/backup_impl_restore_backup.go:337] ["executeRestoreBackupTask start"] [backup_name=backup_2024_04_28_18_12_36_634313588] [backupBucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588]
[2024/04/29 20:13:17.674 +00:00] [INFO] [core/backup_impl_restore_backup.go:389] ["start restore"] [db_name=default] [collection_name=track_fingerprints] [backupBucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588]
[2024/04/29 20:13:17.674 +00:00] [INFO] [core/backup_impl_restore_backup.go:416] ["collection schema"] [fields="[{\"ID\":100,\"Name\":\"primary_id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\
":0},{\"ID\":101,\"Name\":\"track_id\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0},{\"ID\":102,\"Name\":\"segment_id\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataTy
pe\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0},{\"ID\":103,\"Name\":\"fingerprint\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false
,\"ElementType\":0}]"]
[2024/04/29 20:13:17.704 +00:00] [INFO] [core/backup_impl_restore_backup.go:462] ["create collection"] [database=default] [collectionName=track_fingerprints] [hasPartitionKey=false]
[2024/04/29 20:13:17.704 +00:00] [INFO] [core/backup_impl_restore_backup.go:564] ["start restore partition"] [backupCollectionName=track_fingerprints] [targetDBName=default] [targetCollectionName=track_fingerprints] [partition=_default]
[2024/04/29 20:13:17.708 +00:00] [INFO] [core/backup_impl_restore_backup.go:610] ["create partition"] [collectionName=track_fingerprints] [partitionName=_default]
[2024/04/29 20:13:17.708 +00:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588] [partitionID=448284511590680146] [groupId=448358444570611282]
[2024/04/29 20:13:17.724 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611282/,]"] [endTime=449
400795561984]
[2024/04/29 20:13:17.798 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229561341] [state=0] [state="{\"ID\":0,\"State\":0,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1
714421597}"] [progress=0] [currentTimestamp=1714421597] [lastUpdateTime=1714421597]
[2024/04/29 20:13:22.805 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229561341] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"10\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421597}"] [progress=10] [currentTimestamp=1714421602] [lastUpdateTime=1714421597]
[2024/04/29 20:13:27.808 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229561341] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"40\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421597}"] [progress=40] [currentTimestamp=1714421607] [lastUpdateTime=1714421602]
[2024/04/29 20:13:32.812 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229561341] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":281810,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"70\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Create
Ts\":1714421597}"] [progress=70] [currentTimestamp=1714421612] [lastUpdateTime=1714421607]
[2024/04/29 20:13:37.816 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229561341] [state=6] [state="{\"ID\":0,\"State\":6,\"RowCount\":281810,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"100\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Creat
eTs\":1714421597}"] [progress=100] [currentTimestamp=1714421617] [lastUpdateTime=1714421612]
[2024/04/29 20:13:37.816 +00:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588] [partitionID=448284511590680146] [groupId=448358444570611636]
[2024/04/29 20:13:37.830 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611636/,]"] [endTime=449
400795561984]
[2024/04/29 20:13:37.912 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229843275] [state=0] [state="{\"ID\":0,\"State\":0,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1
714421617}"] [progress=0] [currentTimestamp=1714421617] [lastUpdateTime=1714421617]
[2024/04/29 20:13:42.915 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229843275] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"10\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421617}"] [progress=10] [currentTimestamp=1714421622] [lastUpdateTime=1714421617]
[2024/04/29 20:13:47.920 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229843275] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"40\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421617}"] [progress=40] [currentTimestamp=1714421627] [lastUpdateTime=1714421622]
[2024/04/29 20:13:52.924 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229843275] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":282616,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"70\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Create
Ts\":1714421617}"] [progress=70] [currentTimestamp=1714421632] [lastUpdateTime=1714421627]
[2024/04/29 20:13:57.927 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495229843275] [state=6] [state="{\"ID\":0,\"State\":6,\"RowCount\":282616,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"100\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Creat
eTs\":1714421617}"] [progress=100] [currentTimestamp=1714421637] [lastUpdateTime=1714421632]
[2024/04/29 20:13:57.927 +00:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588] [partitionID=448284511590680146] [groupId=448358444570613853]
[2024/04/29 20:13:57.938 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570613853/,]"] [endTime=449
400795561984]
[2024/04/29 20:13:57.984 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495230125999] [state=0] [state="{\"ID\":0,\"State\":0,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1
714421637}"] [progress=0] [currentTimestamp=1714421637] [lastUpdateTime=1714421637]
[2024/04/29 20:14:02.987 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495230125999] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"10\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421637}"] [progress=10] [currentTimestamp=1714421642] [lastUpdateTime=1714421637]
[2024/04/29 20:14:07.990 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495230125999] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"40\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714421637}"] [progress=40] [currentTimestamp=1714421647] [lastUpdateTime=1714421642]
[2024/04/29 20:14:12.996 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495230125999] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":280831,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"70\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Create
Ts\":1714421637}"] [progress=70] [currentTimestamp=1714421652] [lastUpdateTime=1714421647]
[2024/04/29 20:14:17.999 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495230125999] [state=6] [state="{\"ID\":0,\"State\":6,\"RowCount\":280831,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"100\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Creat
eTs\":1714421637}"] [progress=100] [currentTimestamp=1714421657] [lastUpdateTime=1714421652]
...
... more bulk inserts. redacted ...
...
[2024/04/29 23:58:39.829 +00:00] [INFO] [core/backup_impl_restore_backup.go:822] [getBackupPartitionPaths] [bucketName=kube-dev-milvusobjectstoragebucketb003b84a-pxwjd2yuz1wi] [backupPath=backup/backup_2024_04_28_18_12_36_634313588] [partitionID=448284511590680146] [groupId=448358444570639072]
[2024/04/29 23:58:39.840 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570639072/,]"] [endTime=449
400795561984]
[2024/04/29 23:58:39.881 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495421563624] [state=0] [state="{\"ID\":0,\"State\":0,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1
714435119}"] [progress=0] [currentTimestamp=1714435119] [lastUpdateTime=1714435119]
[2024/04/29 23:58:44.885 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495421563624] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"10\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714435119}"] [progress=10] [currentTimestamp=1714435124] [lastUpdateTime=1714435119]
[2024/04/29 23:58:49.888 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495421563624] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"40\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":
1714435119}"] [progress=40] [currentTimestamp=1714435129] [lastUpdateTime=1714435124]
[2024/04/29 23:58:54.892 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495421563624] [state=2] [state="{\"ID\":0,\"State\":2,\"RowCount\":281153,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"70\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Create
Ts\":1714435119}"] [progress=70] [currentTimestamp=1714435134] [lastUpdateTime=1714435129]
[2024/04/29 23:58:59.895 +00:00] [INFO] [core/backup_impl_restore_backup.go:766] ["bulkinsert task state"] [id=449401495421563624] [state=6] [state="{\"ID\":0,\"State\":6,\"RowCount\":281153,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"100\"},\"CollectionID\":0,\"SegmentIDs\":null,\"Creat
eTs\":1714435119}"] [progress=100] [currentTimestamp=1714435139] [lastUpdateTime=1714435134]
[2024/04/29 23:58:59.896 +00:00] [INFO] [core/backup_impl_restore_backup.go:579] ["finish restore partition"] [backupCollectionName=track_fingerprints] [targetDBName=default] [targetCollectionName=track_fingerprints] [partition=_default]
[2024/04/29 23:58:59.896 +00:00] [INFO] [core/backup_impl_restore_backup.go:360] ["finish restore collection"] [db_name=default] [collection_name=track_fingerprints]

Answered by wayblink

May 14, 2024

@schuberttobias Hi, please try the latest version https://github.com/zilliztech/milvus-backup/releases/tag/v0.4.13-rc1

View full answer

yhmo · 2024-05-02T08:30:07Z

yhmo
May 2, 2024
Collaborator

"Restore" calls bulkinsert interface to import data files. Each data node executes task one by one. Multiple data nodes can execute tasks parallelly.

The log shows a task with rowcount=281153 takes 20 seconds, from 23:58:39 to 23:58:59. So, with one data node, 200M rows will takes about 14227 seconds = 237 minutes.

Try use multiple data nodes to restore.

2 replies

schuberttobias May 2, 2024
Author

Hi @yhmo,

many thanks for your response!

Please correct me if I'm wrong, but my understanding was: Milvus Backup only issues bulkinsert tasks serially one by one, regardless of the number of data nodes. Wasn't this supposed to be changed by the unmerged PR zilliztech/milvus-backup#250?

I tested with 4 data nodes and had no speed up. Do you have an idea?

Milvus backup tool log

Don't the logs show that bulkinsert is issued serially?

[2024/04/29 20:13:17.724 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611282/,]"] [endTime=449400795561984]

... backup tool waits until bulk insert done before issuing next bulk insert ...

[2024/04/29 20:13:37.830 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611636/,]"] [endTime=449400795561984]

... backup tool waits until bulk insert done before issuing next bulk insert ...

[2024/04/29 20:13:57.938 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570613853/,]"] [endTime=449400795561984]

Number of folders/files

backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/ holds 685 immediate subfolders and in total 5.400 files. The "files" arguments of bulkinsert in the log each has 18 files.

yhmo May 2, 2024
Collaborator

In milvus server, each data node executes bulkinsert tasks serially.
The client can send many tasks in a short time, and then check the tasks state by interval.

backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/ is a partition's folder, the partition has 685 segments. In the server-size, I am not sure, maybe each segment is treated as a sub-task. Averagely each segmegt has 300k rows, each task takes about 20 seconds to process.

If only one data node in server-side, the tasks are executed one by one.
If there are 5 data nodes in server-side, I think it can execute 5 tasks at the same time.

xiaofan-luan · 2024-05-05T11:22:31Z

xiaofan-luan
May 5, 2024
Maintainer

@bigsheeper
can you help on this? for the new bulkinsert job, there is no need to keep one job per node?
So i'm assuming restore could also be much faster?

0 replies

bigsheeper · 2024-05-07T03:36:55Z

bigsheeper
May 7, 2024
Collaborator

Milvus 2.4.0+ supports importing multiple segments concurrently per node. However, based on the logs, it appears that the milvus-backup tool restores segments sequentially. To leverage import concurrency in the datanode to the fullest extent, I think we should restore by the partition binlog prefix instead of the segment binlog prefix. @wayblink
cc @xiaofan-luan @yhmo

[2024/04/29 20:13:17.724 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611282/,]"] [endTime=449400795561984]

... backup tool waits until bulk insert done before issuing next bulk insert ...

[2024/04/29 20:13:37.830 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570611636/,]"] [endTime=449400795561984]

... backup tool waits until bulk insert done before issuing next bulk insert ...

[2024/04/29 20:13:57.938 +00:00] [INFO] [core/backup_impl_restore_backup.go:723] ["execute bulk insert"] [db=default] [collection=track_fingerprints] [partition=_default] [files="[backup/backup_2024_04_28_18_12_36_634313588/binlogs/insert_log/448284511590680145/448284511590680146/448358444570613853/,]"] [endTime=449400795561984]

11 replies

bigsheeper May 14, 2024
Collaborator

@schuberttobias I'm sorry for any inconvenience caused. This issue requires a change in the way the backup tool uses import.
@wayblink Could you please help to follow up on this issue?

wayblink May 14, 2024

@schuberttobias Hi, concurrent restore is already supported. I will look into your issue today. Will tell you ASAP.

wayblink May 14, 2024

@schuberttobias We will optimize the behavior to support segment level concurrency. The optimistic estimate time is tomorrow.

wayblink May 14, 2024

@schuberttobias Hi, please try the latest version https://github.com/zilliztech/milvus-backup/releases/tag/v0.4.13-rc1

Answer selected by schuberttobias

schuberttobias May 15, 2024
Author

@wayblink Thank you. I will try and report back here.

schuberttobias May 16, 2024
Author

@wayblink I'm currently testing restores with v0.4.13-rc1 and it looks very promising! I will report back with additional information.

schuberttobias · 2024-05-17T09:27:53Z

schuberttobias
May 17, 2024
Author

Test results Milvus Backup Tool v0.4.13-rc1

I've repeated the tests from #32750 (comment) using Backup Tool v0.4.13-rc1.
The fastest restore configuration now took only <5 minutes!

Thanks to everyone involved for your help and the fix!

Cluster

Milvus Cluster v2.4.1 on EKS with 4x AWS c6i.4xlarge
Nodes:
- Proxy = 4
- Query = 8
- Data = 8
- Index = 8

Data set

Same as in #32750 (comment) plus some new vectors. About 207 mio. vectors. 739 segments.

Restore duration

backup.parallelism.restoreCollection: 8

Duration: 30 minutes
Speed up compared to Backup Tool v0.4.12 in Milvus backup tool: How to speed up restore of 200 million vectors? #32750 (comment): 7,5x (=225 minutes / 30 minutes).

backup.parallelism.restoreCollection: 32

Duration: 9 minutes
Speed up: 25x (= 225 / 9)

backup.parallelism.restoreCollection: 739 (= number of segments)

Duration: 3,65 minutes
Speed up: ca. 62x (=225 / 3,65).

Index creation duration

Index created after restore was done.
Duration: 70 minutes.
CPU utilization of EKS Nodes: 100%. Could probably be sped up by scale out/up.

0 replies

xiaofan-luan · 2024-05-18T14:39:22Z

xiaofan-luan
May 18, 2024
Maintainer

nice!
seems to be a great improved!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Milvus backup tool: How to speed up restore of 200 million vectors? #32750

{{title}}

Replies: 5 comments 13 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Milvus backup tool: How to speed up restore of 200 million vectors? #32750

schuberttobias Apr 30, 2024

Problem

Questions

Details

Cluster

show collection

Milvus backup tool config

Milvus backup log

Replies: 5 comments · 13 replies

yhmo May 2, 2024 Collaborator

schuberttobias May 2, 2024 Author

Milvus backup tool log

Number of folders/files

yhmo May 2, 2024 Collaborator

xiaofan-luan May 5, 2024 Maintainer

bigsheeper May 7, 2024 Collaborator

bigsheeper May 14, 2024 Collaborator

wayblink May 14, 2024

wayblink May 14, 2024

wayblink May 14, 2024

schuberttobias May 15, 2024 Author

schuberttobias May 16, 2024 Author

schuberttobias May 17, 2024 Author

Test results Milvus Backup Tool v0.4.13-rc1

Cluster

Data set

Restore duration

backup.parallelism.restoreCollection: 8

backup.parallelism.restoreCollection: 32

backup.parallelism.restoreCollection: 739 (= number of segments)

Index creation duration

xiaofan-luan May 18, 2024 Maintainer

schuberttobias
Apr 30, 2024

Replies: 5 comments 13 replies

yhmo
May 2, 2024
Collaborator

schuberttobias May 2, 2024
Author

yhmo May 2, 2024
Collaborator

xiaofan-luan
May 5, 2024
Maintainer

bigsheeper
May 7, 2024
Collaborator

bigsheeper May 14, 2024
Collaborator

schuberttobias May 15, 2024
Author

schuberttobias May 16, 2024
Author

schuberttobias
May 17, 2024
Author

xiaofan-luan
May 18, 2024
Maintainer