Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrent builds on gorgona fail becaues of running out of memory #1343

Open
mpol opened this issue Apr 21, 2022 · 1 comment
Open

Concurrent builds on gorgona fail becaues of running out of memory #1343

mpol opened this issue Apr 21, 2022 · 1 comment

Comments

@mpol
Copy link
Contributor

mpol commented Apr 21, 2022

IIS tests on gorgona are sometimes killed (see eg. Crashed tests in https://gorgona.ocean.icm.edu.pl/job/openaire/job/iis/job/master/25/consoleFull), probably by the OOM killer. This happened recently during concurrent IIS builds, but may also happen when other jobs run concurrently with IIS builds. The default for -Xmx on Java 8 is 25% of memory, we can decrease it and be more memory friendly, but other concurrent builds can still cause problems.

Additionally it seems that in the current configuration crashed tests are detected only because of the interaction with the Sonar JaCoCo sensor - Sonar finds empty jacoco output and fails. The junit archiving plugin eg. does not see them, and with-Dmaven.test.failure.ignore apparently crashes are ignored too.
@mpol
Copy link
Contributor Author

mpol commented Apr 25, 2022

Reducing -Xmx is an option but from local testing it seems that the tests do not produce enough garbage to fill what they get by default anyway.
And today we got failures again when multiple builds of different branches were running concurrently.
To decrease pressure on the build machine we should probably start with adding disableConcurrentBuilds to the Jenkinsfile.

mpol added a commit that referenced this issue Apr 25, 2022
@mpol
Disable concurrent builds to reduce memory usage
c62add1 
As described in the issue #1343 builds get killed by the OOM killer
sometimes.

Add disableConcurrentBuilds to the build options to reduce
memory usage whem multiple builds are triggered.
@mpol mpol mentioned this issue Apr 25, 2022
Merged
mpol added a commit that referenced this issue Apr 25, 2022
@mpol
Disable concurrent builds to reduce memory usage
33869d0 
As described in the issue #1343 builds get killed by the OOM killer
sometimes.

Add lock('iis-build') to the build options to reduce
memory usage whem multiple builds are triggered.
As described eg. in https://issues.jenkins.io/browse/JENKINS-35359
the disableConcurrentBuilds option can only prevent concurrent builds
of a single branch and using locks is the recommended workaround
for serializing all the builds in a multibranch pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mpol