Compiling MAME with flatpak-builder makes my buildserver crash

[XargonWan]

Greetings,

I'm trying to compile MAME with flatpak-builder as you do in the original manifest, but it makes our buildserver crash after 50 minutes of compilation or such: the server just becomes unresponsive (even over ssh) and the build run fails.
Here is our RetroDECK-MAME repo.

Our buildserver specs:

CPU: Intel Xeon E3-1230v6 - 4c/8t - 3.5 GHz/3.9 GHz
RAM: 16 GB ECC 2133 MHz

I tried to set a lower number of jobs, even tried with -j 1 but it doesn´t seem to affect our issue.
Any help?

If you're wondering why it's not in the main RetroDECK manifest is because this issue and because MAME takes long to compile and a fresh compilation is not needed unless on version bump.

[startaq]

Well, any logs? 16 GB RAM should be okay for -j 4. Is there swap space on the server?

[rb6502]

Yeah, a log of the crash would be useful. And build times don't have to be bad - I can do a clean full build of MAME in exactly 8 minutes on my new Apple M3 Max with -j16 and 64 GB of RAM. I suspect the RAM helps more than the number of cores, because my 32 GB Ryzen 3900X is significantly slower.

[XargonWan]

How can I generate useful logs?
What I got from the GitHub Action Runner is just a truncated log where it's building until the server dies.

[galibert]

Please try compiling with clang and, importantly, linking with lld. The memory usage is significantly lower and the link speed goes from tens of minutes to seconds.

[XargonWan]

I added the said lines but after 6h of compilation it just stopped.

  - name: mame
    buildsystem: simple
    build-options:
      no-debuginfo: true
      strip: true
      env:
        CC: clang
        CXX: clang++
        LD: lld
    build-commands:
      - make
          -j echo $(nproc)
          CFLAGS+=-Wno-error=restrict
          USE_QTDEBUG=0 
          USE_WAYLAND=1
          USE_SYSTEM_LIB_FLAC=1
          SDL_INI_PATH='$$HOME/.APP_NAME;/app/share/APP_NAME/ini'
          LDOPTS=-Wl,-s
     [...]

Full manifest

EDIT: now I tried to put that flags directly inside the make command unstead of the build-options.

[XargonWan]

So I tried with the inline options, however the issue is that agter 6 hours our self-hosted builder stops. I tried to make it longer but that option seems to be ignored, I will check on that.

However is that normal that a MAME flatpak build takes over than 6 hours? I think something's wrong.

[rb6502]

6 hours is definitely excessive for a MAME build unless it's got a really tiny amount of RAM (4GB or less).

[XargonWan]

I got 16GB, what could possibly be wrong?

[XargonWan]

I tried to replicate the entire flatpak manifest as-it-is in this branch but github actions after 40 minutes stops with this error:

The self-hosted runner: mame-runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

If I try to ssh in the buildserver it hangs, I have to reboot via its control panel.

[startaq]

This isn't actually an issue with MAME, but I saw the following in your flatpak manifest: -j ${FLATPAK_BUILDER_N_JOBS}. The Flatpak documentation says:

FLATPAK_BUILDER_N_JOBS
The number of jobs that flatpak-builder would normally use for make -j. Defaults to ncpus unless the module disabled parallel make.

This probably means 8 in your case, which might be too much for a system with only 16 GB RAM (and no swap space? you didn't answer the question).

You said changed this to -j 1. Are you sure you did that correctly and it was picked up? Note that -j 1 might not be enough to build MAME within the time limit, so you should aim for something between 2 and 4.

[cuavas]

For what it’s worth, I build MAME on Linux virtual machines with 6GB RAM with -j5. It does hit swap, but it completes.