How to keep primary key unchanged when copying one collection from Milvus to another with auto-id on

[heibaidaolx123]

Hi,
if the collection has a schema with auto-id set to True, is there a way to keep primary key unchanged when copying one collection from Milvus to another?
I know milvus-backup might be a choice, but I just want to copy one collection, not all of them. And according to the doc, milvusdm is perfect for my needs, but only milvus 1.x are supported.
Any idea?
Thanks.

[heibaidaolx123]

It seems that when auto_id is set to True, upsert will fail if I want to update an item indexed by id.
So auto_id conflicts with data that id is manually-set, in all cases, including when updating is wanted.
Is this by design?

[yhmo]

In fact, by using milvus-backup restore, the auto-generated ids are copied into the new collection without any change.

Yes, upsert cannot support collections with auto_id=true, this is as-designed in current versions.

[heibaidaolx123]

Got it, thanks a lot.

[heibaidaolx123]

@yhmo hi, yihua,
according to milvus-backup doc, bulkinsert is used when restoring from backup.
I tested bulinsert with auto-id, by examples/example_bulkinsert_json.py (had changed auto_id=False to True), and it failed with the error blow:
{'failed_reason': "the primary key 'id_field' is auto-generated, no need to provide: importing data failed", 'progress_percent': '0'}
So, maybe milvus-backup restore doesn't work as well for auto_id collection?
Any idea to walkaround?

[yhmo]

The example tests bulkinsert interface. The bulkinsert interface has its validation for input.
If auto_id=True, not allow to input data for pk in the JSON files.
If auto_id=false, users must provide pk values in the JSON files.
It is not a problem of milvus-backup.

[heibaidaolx123]

To update, I tried the steps in milvus-backup doc:

./milvus-backup create -n test_backup -c db_c61caec2a5f64a9ca5b2869a3cb0843f
./milvus-backup restore -n test_backup -s _test --restore_index

where db_c61caec2a5f64a9ca5b2869a3cb0843f is a collection with auto-id.
It succeeded somehow, but the ids, which is the primary field, for the 2 collections did not match.

[yhmo]

No matter the auto_id is true or false, after restore, the two collections have the same ids. You can use the same vector to do a search, the returned result have the same ids.

[heibaidaolx123]

To explain, in my application scenario, I want return the ID to user, and then, user can get/query/update by that ID directly without manipulating IDs. And auto-id is perfect for this.
Lets say now there's too many collections in one Milvus instance, for a reason, and we need to split a half of users' collections to another Milvus instance, and we do it by backup and restore.
I need to ensure that users, directed to the new Milvus instance, can use the IDs as brefore. The ids need to be unchaned during backup-restore.
IMO, this is a common case. Is it possible for now? And will it in future?

[yhmo]

No matter the auto_id is true or false, after restore, the two collections have the same ids.
I have verified this in my local.

[heibaidaolx123]

I reran my test, and comfirmed that after restore, the ids are not the same between the 2 cols.
I just had one item in both cols, and the ids are 448910346970203124 and 448910346971974413.
My env:

python -c "import pymilvus;print(pymilvus.__version__)"
2.4.0

./milvus-backup -v
0.4.12 (Built on 2024-03-13T10:20:58Z from Git SHA 7b7c7b10c51e583085323cac86a5d4e16e9bf832)
milvus-backup version 0.4.12 (Built on 2024-03-13T10:20:58Z from Git SHA 7b7c7b10c51e583085323cac86a5d4e16e9bf832)

./milvus-backup check
0.4.12 (Built on 2024-03-13T10:20:58Z from Git SHA 7b7c7b10c51e583085323cac86a5d4e16e9bf832)
config:backup.yaml
[2024/04/07 19:40:25.479 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/04/07 19:40:25.479 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/04/07 19:40:25.486 +08:00] [INFO] [storage/minio_chunk_manager.go:151] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
Succeed to connect to milvus and storage.
Milvus version: v2.4.0-rc.1
Storage:
milvus-bucket: a-bucket
milvus-rootpath: files
backup-bucket: a-bucket
backup-rootpath: backup

Is there anything I missed?

[yhmo]

So you are using v2.4.0.
My milvus is v2.3, the primary keys are correctly copied to the new collection. I think this is a regression of v2.4, we will fix it in the next minor version of v2.4.
If you want to keep pk unchanged, you can use a milvus v2.3.

[heibaidaolx123]

Thanks.