Background operations after the index has been built

[ronbarhen]

Hello,

I have a database of 500 millions 96 float32 features records.
I use a script which is very close to the script "example_bulkinsert_numpy.py" in pymilvus in order to create the index and insert the data into it.
After about 4 hours when the index is built and after "utility.wait_for_index_building_complete()" returns and "collection.load(_refresh = True)" returns, the index is searchable but the search speed is very slow.
Moreover, if I look at htop, after the index is built and when no search is performed, I can see that the CPU is heavily used in the background.
The situation is even worse if I define a segment size of 128GB, then, you have to wait 3-4 hours in order for the search speed to be OK.
My questions are:

1. What does Milvus perform in the background after the index has been built?
2. Is there a way to know when those operations are finished programmatically?
3. What is the maximal segment size supported by Milvus?

Thanks!
Ron

[xiaofan-luan]

segment size is too large. usually what we recommend is 2GB - 4GB. otherwise index build will be very slow.

[xiaofan-luan]

utility.wait_for_index_building_complete actually means index build down.
But I guess some compaction happens and another round of index task triggered.

[xiaofan-luan]

That's we use index pool on zilliz cloud. it scales automatically with more tasks

[ronbarhen]

Is there a way to synchronously perform the compactions or the other tasks that are performed in the background so that I can measure how long time they took and know when I can start searching?

[xiaofan-luan]

Not yet, but you can try to trigger compaction manually.

Anyway, the key is keep segment in a reasonable size. Most likely you dind't need to worry about background tasks.
And it always better to deploy index/data node to a seperate pod