memory usage of data coordinator keeps increasing #31226
-
|
Hello, i have a question about memory usage of data coordinator We are running a data pipeline for milvus (v2.3.10) that repeats below process.
p.s segment count of single collection is about 1700 ~ 1800 data pipeline starts at everyday 4 AM, and after repeating this for week, it seems like the memory usage of data coordinator is gradually increasing
There are some questions i want to ask.
below is log and pprof for data coordinator milvus log link: https://sendanywhe.re/L6K266NZ |
Beta Was this translation helpful? Give feedback.
Replies: 11 comments 16 replies
-
The material you provided is rich, firstly let us read the log and profile to investigate. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like the cluster has: |
Beta Was this translation helpful? Give feedback.
-
|
Yes our k8s cluster has limit that max limit of core and memory of pod is 8c15g. So we had to increase number of query node to get sufficient memory |
Beta Was this translation helpful? Give feedback.
-
|
Did you have a monitoring service to observe the cluster? |
Beta Was this translation helpful? Give feedback.
-
|
yes. below is capture of datacoord metric of past 7 days.
I think it's strange that collection num (milvus_datacoord_collection_num) keeps increasing even if i dropped collection. Can this be a possible reason? |
Beta Was this translation helpful? Give feedback.
-
|
I have another question. to check below question, i manually triggered OOM of active datacoord using stress-ng command.
But it seems like two datacoord pods are keep switching active-standby forever, and not working normally.
below is log file |
Beta Was this translation helpful? Give feedback.
-
|
we tried below step, but not resolved.
|
Beta Was this translation helpful? Give feedback.
-
|
we version up to 2.3.11, but not resolved. |
Beta Was this translation helpful? Give feedback.
-
|
is there a reason we have so many segments? 
Did we do manual flush or num_entities? |
Beta Was this translation helpful? Give feedback.
-
|
In our side, we didn't do manual flush or call num_entities. below is our collection insert process
I thought number or dropped segment increase is due to partition key is enabled and compaction occurs during bulk insert. Is our segment number metric is abnormal and cause of datacoord's large memory usage? |
Beta Was this translation helpful? Give feedback.
-
|
After reading the log, I noticed some strange points:
Is there another cluster running with the same etcd server? |
Beta Was this translation helpful? Give feedback.
-
|
Let me explain our situation. |
Beta Was this translation helpful? Give feedback.
-
in grafana metric, It seems like import request are received to proxy
I thought upper logs means that all data node / index node are busy to process bulk insert / index create process, and retry when node become available. below is cpu / memory usage of data / index nodes of past week
|
Beta Was this translation helpful? Give feedback.
-
|
Did you change the log level of milvus? Seems all INFO level logs of rootcoord are missed after 2024/03/05 14:42:02 |
Beta Was this translation helpful? Give feedback.
-
|
All the logs of the 4 data nodes are from 2024/03/13 09:20, I didn't know what happened inside the 4 data nodes. |
Beta Was this translation helpful? Give feedback.
-
|
I checked our helm file, and found that log level was set to warn we will try to upgrade to 2.3.11 and modify log level to info image:
all:
tag: v2.3.10-root
attu:
image:
tag: v2.3.8-root
log:
level: "warn"
... |
Beta Was this translation helpful? Give feedback.
-
|
From the pprof
UpdateDropChannelSegmentInfo -> Should we do clone? also, I believe the question could be:
|
Beta Was this translation helpful? Give feedback.
-
|
@yiwangdr |
Beta Was this translation helpful? Give feedback.
-
|
@xiaofan-luan That looks like the cause. We need logs to confirm it. @Jung-JongHyuk is it possible for you to get the datacoord log that can cover the "drop-then-spike" period (e.g. 03/11 14:00-18:00). Meanwhile we will try to reproduce it. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for help. It's hard to share past datacoord log because we already reinstalled milvus, and log level was warn. I have one more question. I saw an issue about gc of dropped collection (#31306) |
Beta Was this translation helpful? Give feedback.
-
|
could be. If you get any luck to reproduce, please connect me @ xiaofan.luan@zilliz.com and I would like to have a short discussion on zoom if possible. thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@Jung-JongHyuk if you'd like to verify if the collections are cleaned up, please try birdwatcher(https://github.com/milvus-io/birdwatcher). |
Beta Was this translation helpful? Give feedback.
-
|
Hello, after some trial to reproduce, we found the root cause. when milvus do compaction / drop collection, segment files were deleted in object storage but it's directory remained. when we reinstalled milvus with minio as object storage, both scanning time and datacoord memory usage kept stable
We will keep watching to check whether is there another leak in other components. Thanks for helping us! |
Beta Was this translation helpful? Give feedback.
-
|
@Jung-JongHyuk Thanks for the update. Good to know that minio config works well. What was the old config? Could you provide the config file? We'd like to reproduce it and fix the issue. Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
@xiaofan-luan @yiwangdr sorry for late reply. When we install milvus, we used custom object storage platform (private and non-standard platform which only used in our company) so when milvus delete files in object storage (in garbage collector of datacoord), empty directory remained in our object storage (this wasn't bug of milvus, but because our object storage didn't work same as s3 / minio when delete object) those empty directory are included in scan target in ListWithPrefix method (searching files using bfs), so scan time became longer and longer, and queue size in ListWithPrefix method became bigger. It was our fault to use non-standard object storage for milvus. |
Beta Was this translation helpful? Give feedback.
-
|
we are actually working on a improve the scan efficiency. If I understand correctly you are actually using a file system rather than object storage? We believe it would be great if we can implement a fs chunk manager |
Beta Was this translation helpful? Give feedback.
Hello, after some trial to reproduce, we found the root cause.
The main reason was that the object storage we used was not 100% compatible with s3 / minio.
when milvus do compaction / drop collection, segment files were deleted in object storage but it's directory remained.
those empty directories included to scan target in garbage collector of datacoord, so time spent for scanning storage kept increasing like below log