Question about other data field and memory spike

[hjlee9182]

Hi I have a question :)

Milvus verson. 2.3.10
collection info

• 1M rows
• 768 dim
• length 9800 varchar

1. When I load a collection, a memory spike occurs. Why does that happen?

2. When I load the same collection, the size of the memory spike varies. What could be the reason?
   

3. It seems like the amount of memory loaded varies when a collection contains fields of different data types, such as integers, floats, and varchars, in addition to embeddings. Is this because the different data types also get loaded into memory, affecting the overall size?

4. If the third question is correct, would the sizes for each data type be as follows?

bool: 1 byte
int8: 1 byte
int16: 2 bytes
int32: 4 bytes
int64: 8 bytes
float : 4 bytes
double : 8 bytes
varchar: length of string * 1bytes

[xiaofan-luan]

Hi I have a question :)

Milvus verson. 2.3.10 collection info

• 1M rows
• 768 dim
• length 9800 varchar

1. When I load a collection, a memory spike occurs. Why does that happen?
2. When I load the same collection, the size of the memory spike varies. What could be the reason?
   
3. It seems like the amount of memory loaded varies when a collection contains fields of different data types, such as integers, floats, and varchars, in addition to embeddings. Is this because the different data types also get loaded into memory, affecting the overall size?
4. If the third question is correct, would the sizes for each data type be as follows?

bool: 1 byte
int8: 1 byte
int16: 2 bytes
int32: 4 bytes
int64: 8 bytes
float : 4 bytes
double : 8 bytes
varchar: length of string * 1bytes

when load happens, there are many concurrency file handling and copy, that is why it nead actual memory to load. We recommend to leave at least 2GB of memory buffer for load.

yest you can estimate the memory usage like that, this is not accurate (usually more than that with some overhead)

[hjlee9182]

Is the size of the overhead not fixed and different for each data type and data?

[yhmo]

In milvus, string is encoded by utf8, each character occupies 1 ~ 4 bytes(1 byte for ANSI and 2 ~ 4 bytes for characters in different languages). So, for English, the size of varchar is length of string * 1bytes. But for Chinese or Janpanese, the size of varchar is length of string * 3bytes or length of string * 4bytes