Out of memory (exit code 137) when building Scala 3-native artifacts in tapir

[adamw]

Since some time, most of the tapir builds fail with Error: Process completed with exit code 137. when building Scala 3 native artifacts.

This is unfortunately not deterministic, as about 20% or so builds pass (so retrying the build helps). Also, Scala 2 native builds just fine.

I'm not sure how to debug this - there are some warnings on the console:

[warn] In the last 9 seconds, 7.291 (85.1%) were spent in GC. [Heap: 1.37GB free of 4.00GB, max 4.00GB] Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.

But we're already allocating the maximum memory available on a github actions macine.

An example failing build is here: https://github.com/softwaremill/tapir/actions/runs/4751063533/jobs/8439821428

There's a couple of modules built for native, but not that many (core, tests, some integrations, one server interpreter).

Some guidance as to how to debug this would be greatly appreciated :)

[armanbilge]

But we're already allocating the maximum memory available on a github actions macine.

Ubuntu runners have 7 GB (from your warning seems you've maxed out at 4?). If that's not enough, you can use a macOS runner instead which offers 14 GB.

Also, consider setting a concurrency limit.

https://github.com/typelevel/cats/blob/14286c9ef89a652fcff182bbd685f6caf930701e/build.sbt#L68

[adamw]

Well we're using 4GB for the heap, which leaves 3GB for the entire rest (non-heap and the OS). Using anything more than 4GB for the heap makes the build OOM (on the kernel level) with a 100% certainty ;)

We'll try the macos runners, thanks, though that's more of a work-around than a fix. I don't think compiling a bunch of modules using scala-native should require more than 4GB of heap :) Besides, it works for JVM (compiling all modules), JS and native-2, so there's clearly something wrong with native-3.

[adamw]

Btw. we already have concurrency restrictions enabled globally :) https://github.com/softwaremill/tapir/blob/master/build.sbt#L31

[WojciechMazur]

Yes, but it only limits the execution of tests, every task which is a test dependency can still be run in parallel.
It might not be highly visible in the CI with 2 CPU's but with more of them we can easily spot concurrent linkings in Scala.js and Scala Native
I've submitted a PR to Tapir which should mitigate that issue, until I'll manage to optimize the Scala Native builds
softwaremill/tapir#2855

[adamw]

This seems to have solved the problem (at least work around the problem :) ).

Thanks a lot @WojciechMazur !