Out of Memory error

[adamlc]

Hello occasionally I am getting this error in the executor:

Terminating due to java.lang.OutOfMemoryError: Metaspace

I've deployed Terrakube with the standard helm chart and not changed any resource allocations. It's well within the 2GB limit set. Is there any way I tweak this? I'm not too familiar with Java.

[adamlc]

This also leaves the job in a broken state thats pending forever. Even if I cancel it any subsequent jobs are also stuck

2024-02-15 11:59:00.103 WARN 1 --- [ryBean_Worker-3] o.t.api.plugin.scheduler.ScheduleJob : Job 178 is waiting for previous jobs to be completed...

[alfespa17]

If for some reason a job is stuck it will be cancelled after 6 hours.

Other option is go to the database a change the status tu CANCELLED

[adamlc]

The strange thing here is the job is already marked as cancelled and in the DB too

[alfespa17]

Could you check all the jobs for that particular workspace? Maybe some job is pending queue or running? The queue should continue if all the previous jobs are failed or completed state

terrakube/api/src/main/java/org/terrakube/api/plugin/scheduler/ScheduleJob.java

Line 86 in 191689a

Optional<List<Job>> previousJobs = jobRepository.findByWorkspaceAndStatusNotInAndIdLessThan(job.getWorkspace(), Arrays.asList(JobStatus.failed, JobStatus.completed), job.getId());

[alfespa17]

It looks like I will have to add cancelled state to that validation

[adamlc]

Yes I think so, changed the job state to failed and the pending one started right away, thanks :)

[alfespa17]

I think you can fix that changing the resource allocation, check this:

https://github.com/AzBuilder/terrakube-helm-chart?tab=readme-ov-file#42-node-affinity-nodeselector-taints-and-tolerations

[adamlc]

https://www.mastertheboss.com/java/how-to-solve-java-lang-outofmemoryerror-compressed-class-space-error/

[adamlc]

I can see at runtime it's calculated automatically. Maybe just increasing the memory overall would help, but being able to override that may be useful

Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx1387205K -XX:MaxMetaspaceSize=197946K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 2G, Thread Count: 250, Loaded Class Count: 32534, Headroom: 0%)

[alfespa17]

The container is built using the java buildpack there is a feature that automatically allocate the JVM memory based on the available resources for the pod

https://github.com/paketo-buildpacks/java

[alfespa17]

You could do some customization using that Java tools option like in this example:

https://docs.terrakube.io/getting-started/deployment/proxy-configuration

[adamlc]

Yes that looks like it could work, thank you!