[BUG] OpenSearch 1.2.x is consuming more Resident memory on JDK11 as compared to JDK 8 #13623
Comments
reta
changed the title
|
Another report from the forum [1]:
[1] https://forum.opensearch.org/t/opensearch-is-consuming-more-cpu-on-jdk11-as-compared-to-jdk8/19156/2 |
|
One of the major differences between JDK-8 and JDK-11 is the change of the GC: JDK-8 uses parallel GC by default, whereas JDK-11 uses G1. As an experiment, you may try to switch to parallel GC. Also please check out [1] that provides some measurements regarding memory overhead for different JDKs and GCs. [1] https://kstefanj.github.io/2021/11/24/gc-progress-8-17.html |
|
Thanks for the quick response. We are using CMS GC in both JDK8 and JDK 11. |
Understood, CMS is gone in JDK-14 [1], have you tried with G1 (JDK-11) and switched to CMS, or CMS was just something you've been using all the time? |
|
Yes we are using CMS all the time as this was recommended by Opensearch in jvm options |
May you please point out where this recommendation is? AFAIK CMS was superseded by G1 for JDK-11 in |
|
In jam.options CMS is recommended from Jdk8-13 |
|
In our case since we are using CMS in both JDk 8 and JDK11, could you please help in finding the root cause for increase in resident memory on JDK 11 |
Oh right, you are using 1.2.x - this release line is not maintained anymore. I would recommend you enable NTM [1] and compare the memory tracking between JDK-8 and JDK-11. [1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html |
|
Sure, I will post the memory tracking here |
|
JDK8 native memory tracking Native Memory Tracking: Total: reserved=3600098KB, committed=2372054KB
JDK11 native memory tracking Total: reserved=3585904KB, committed=2271872KB
I don't see much difference in between those 2. However difference in resident memory between these 2 is 100-200 MB. Jdk 11 JDK8 |
|
Interesting, Could you please share |
Sure. Here it is;
|
|
|
Oops, did you run |
|
@reta Sorry my bad. I have captured all the data again. JDK 8top command output - 367743 search-user 20 0 9879256 2.4g 32660 S 5.3 5.1 14:06.49 367743 /usr/lib/jvm/openjdk-java8-runtime-amd64/bin/java -Djava.awt.headless=true -XX:+PrintGCDetails -XX:+ NMT Native Memory Tracking: Total: reserved=3605658KB, committed=2388146KB
PMAP 367743: /usr/lib/jvm/openjdk-java8-runtime-amd64/bin/java -Djava.awt.headless=true -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -Xloggc:/var/log/search/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djava.security.manager -Djava.security.policy=/etc/search/conf/search.policy -Dlogging.subcomp=search -Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/etc/opensearch -XX:+UseStringDeduplication -XX:+HeapDumpOnOutOfMemoryE total kB 9928820 2504096 2470912 JDK11top command output NMT Native Memory Tracking: Total: reserved=3611428KB, committed=2300392KB
PMAP total kB 10065128 2759292 2719220 |
|
@reta JDK8<style> </style>
JDK11<style> </style>
Here I can see the allocation of RSS is more on JDK11. |
|
@swapnilsvaidya just got back to this, yeah, it seems like from |
|
I was glancing through a number of articles / reports / posts and there is strong signals that in general JDK-11 memory consumptions is slightly higher than JDK-8, I think it is difficult to attribute that to any particular change (since there are at least 2 releases in between), but this is something to account for. |
peternied
changed the title
peternied
added
Build
reta
changed the title
|
@peternied I have changed 1.3 back to 1.2.x since there is a diff in default GC between these two versions (conf/jvm.options) |
Describe the bug
I have OpenSearch installed as single node.
I have observed that though I have allocated max heap size as 983MB to OpenSearch. The res memory in top command for this process shows as 1.6
JDK 11 top command output.
74745 nsx-sea+ 20 0 6006980 1.5g 25476 S 0.0 6.6 17:02.91 74745 /usr/lib/jvm/openjdk-java11-runtime-amd64/bin/java -Djava.awt.headless=true -Xlog:gc*,safepoint:/va+
However under the same scale if Openasearch is installed using JDK8 then it is consuming around 300 MB less memory.
JDK 8 top command output.
84969 search-user 20 0 5912720 1.3g 0 S 0.0 5.7 154:46.99 84969 /usr/lib/jvm/openjdk-java8-runtime-amd64/bin/java -Djava.awt.headless=true -XX:+PrintGCDetails -XX:+
Could someone please help me to understand what factors needs to be checked to analyze increase in RES memory used by OpenSearch in JDK11.
I checked the heapdump but it is not consuming the complete heap allocated to it.
Related component
Other
To Reproduce
Install OpenSearch 1.2.3 using JDK8 and JDK11 and compare the top command output in terms of memory for OpenSearch process.
Expected behavior
Memory consumption should be same for JDK 8 and JDK11.
Additional Details
Plugins - No Plugins installed
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: