Max segment size in Milvus 2.4

[ronbarhen]

Hi,

I think that the maxSize in the configuration file milvus.yaml has no effect in Milvus 2.4.
It had an important effect in previous versions, but now when I change it, I get always the same build times, recalls and latencies.

Thanks!

[yhmo]

I just verified the maxSize by a local standalone, it works well.
Test steps:

1. mount the milvus.yaml to outseide in the docker-compose.yaml

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.0-rc.1
    ......
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
      - ${DOCKER_VOLUME_DIRECTORY:-.}/milvus.yaml:/milvus/configs/milvus.yaml

2. change the dataCoord.segment.maxSize to 1024(MB) in the milvus.yaml
3. docker-compose up -d to start the standalone
4. insert 200k vectors(dimension=1024) into a collection, the total data size is 200k * 1024 * 4bytes=800MB
5. deploy an Attu to observe, the collection has only 1 sealed segment, 200k/800Mb
   

Then stop the milvus, change the dataCoord.segment.maxSize to 256(MB) and try the steps again. Observe the Attu, there are 3 sealed segments in the collection: 60k/240MB, 60k/240MB, 80k/320MB. This is expected.

[xiaofan-luan]

5M data per segment is not a small number. Why do you think tune the params not work?
For memory index, by default it's 512M

[ronbarhen]

5M data per segment corresponds to a 2G segment and I inserted maxSize = 10G in the milvus.yaml configuration file.

[yhmo]

In the source code of v2.4, I didn't see any logic to limit the upper value of segment.maxSize.

[ronbarhen]

OK - Thanks!

[ronbarhen]

It seems that if the client sends 2GB in the function bulk_insert_columnbased() then segments of size 2GB will be created no matter what is the value of maxSIze (if it is >= 2GB), and if the client sends 4GB (the maximum supported by gRPC), then segments of size 4GB will be created if maxSize >= 4GB.
Maybe it's a bug?

[xiaofan-luan]

5M data per segment corresponds to a 2G segment and I inserted maxSize = 10G in the milvus.yaml configuration file.

5M is more than 3G and index usually take more. Anthing beyond 5G is reasonable if you set max segment size to 5G

[ronbarhen]

But I set maxSize to 10GB, and it was the same when i set it to 32G or 64G.
(the dataset is 96 float32 features)

[xiaofan-luan]

But I set maxSize to 10GB, and it was the same when i set it to 32G or 64G. (the dataset is 96 float32 features)

how did you set the segment size?
did you check the segment size with birdwatcher?

[ron-barhen]

by setting the corresponding maxSize in the milvus.yaml file.
I checked the segment size using Attu.
Thanks!