Cannot build kernel with debug symbol if build box has only 8GB or RAM

[prahal]

What happened?

Linux debug kernel package build abort in dpkg-deb on no space left on device. It turns out that
armbian mount /armbian/.tmp/work-<id> as tmpfs at 99% of the RAM.
But on my box there is only 8GB of RAM and the debug kernel and its debian package directory are bigger than that amount.

LC_ALL=C free -mwh
               total        used        free      shared     buffers       cache   available
Mem:           7.5Gi       5.2Gi       176Mi       497Mi       0.0Ki       2.2Gi       1.6Gi
Swap:           20Gi       9.3Gi        10Gi

[🐳|🔨]   dpkg-deb: building package 'linux-image-edge-rockchip64' in '/armbian/.tmp/work-5afda07d-31f4-497b-b258-6f94f31c9c40/tmp.QJs6FujvRC//linux-image-edge-rockchip64_23.05.0-trunk--6.3.3-S1700-D3037-Pb816-C82c7Hfe66-HK01ba-Vc222-B9c18_arm64.deb'.
[🐳|🔨]   dpkg-deb (subprocess): compressing tar member: pass-through copy error: failed to write (No space left on device)

Just before error, running for i in $(seq 1 100); do date; df -h ; sleep 2 ; done in the build container (docker exec -ti stupefied_dijkstra bash) I see that WORKDIR tmpfs gets full. No way to see that from the docker host it seems.

Mon May 22 13:40:37 UTC 2023
Filesystem      Size  Used Avail Use% Mounted on
overlay         160G  144G   12G  93% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/sdb6       680G  497G  181G  74% /armbian/lib
tmpfs           3.8G   24K  3.8G   1% /tmp/dev/shm
/dev/sdb5       160G  144G   12G  93% /armbian/.tmp
tmpfs           7.5G  704K  7.5G   1% /armbian/.tmp/logs-5afda07d-31f4-497b-b258-6f94f31c9c40
tmpfs           7.5G  7.4G   85M  99% /armbian/.tmp/work-5afda07d-31f4-497b-b258-6f94f31c9c40

How to reproduce?

./compile.sh BOARD=helios64 BRANCH=edge RELEASE=bullseye kernel
with in userpatches an edge kernel config with debug info enabled:

userpatches/linux-rockchip64-edge.config

CONFIG_DEBUG_INFO=y
CONFIG_AS_HAS_NON_CONST_LEB128=y
# CONFIG_DEBUG_INFO_NONE is not set
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
CONFIG_DEBUG_INFO_DWARF5=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set

Branch

main (main development branch)

On which host OS are you observing this problem?

Bullseye

Relevant log URL

https://paste.next.armbian.com/zibewonidu

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Jira ticket: AR-1743

[prahal]

The build process is fine once I comment out the line:

LOG_SECTION="prepare_tmpfs_workdir" do_with_logging prepare_tmpfs_for "WORKDIR" "${WORKDIR}" 

from prepare_host_init in lib/functions/main/start-end.sh.
/armbian/.tmp is part of the non-persistent docker volume.
Is there a requirement to put workdir /armbian/.tmp/work- into a tmpfs? speed?

With the commented-out line, I get a linux image with a data.tar of 3.2GB on its own. So with the copy dpkg-deb imports this makes 6.4GB of minimum required memory for the build system for a debug kernel.

[rpardini]

Yep, we could add a way to skip tmpfs for those specialized debug builds.
The build system is trying to use tmpfs for speed and is focusing on the standard, non-debug, kernels.
Let's see if we can find anyone interested in supporting this for 23.08+

[prahal]

Building the debug kernel is broken with armbian default CONFIG_MODULE_COMPRESS_XZ=y.
One has to set CONFIG_MODULE_COMPRESS_NONE=y

The kernel scripts/package/buildded strip and resigning the modules after the modules_install stage.
But one cannot strip and resign from a compressed module (and modules_install apply the compression)
I believe upstream is still broken in that regard even after the recent rework of this script.
But I see no way to fix that (

Plus armbian instance of this script had a bug in that it was doing "make clean" before copying vmlinux into the debug build directory (thus the debug files like vmlinux were already cleaned and their copy failed). This is for the BUILD_DEBUB in the non-native case:

                        # Clean up the executables that are left over from
                        # cross-compilation for a different host architecture.
                        (
                                cd $srctree
                                make M=scripts clean
                        )
                        deploy_kernel_headers $kernel_headers_dir
                        create_package $kernel_headers_packagename $kernel_headers_dir "headers"
(...)
if [ -n "$BUILD_DEBUG" ]; then
       # Build debug package
       # Different tools want the image in different locations
       # perf
       mkdir -p $dbg_dir/usr/lib/debug/lib/modules/$version/
       cp vmlinux $dbg_dir/usr/lib/debug/lib/modules/$version/
       # systemtap
       mkdir -p $dbg_dir/usr/lib/debug/boot/
       ln -s ../lib/modules/$version/vmlinux $dbg_dir/usr/lib/debug/boot/vmlinux-$version
       # kdump-tools
       ln -s lib/modules/$version/vmlinux $dbg_dir/usr/lib/debug/vmlinux-$version
       create_package "$dbg_packagename" "$dbg_dir"
fi

back then I made a modification of the script to move the debug package creation before the make clean.
This is not necessary anymore since armùbian-next rework removed this code.

So I believe you will find nobody that was building armbian kernel with debug info as that was broken.

The thing is with SSD, with the tmpfs or without it seems to make no difference speed wise for the work directory.
We copy from the disk to the tmpfs and back. I believe I should benchmark the difference between tmpfs and without. Maybe also I do not see much difference because I am CPU constrained. You might have beast I cannot compete with.

[rpardini]

Building the debug kernel is broken with armbian default CONFIG_MODULE_COMPRESS_XZ=y.
One has to set CONFIG_MODULE_COMPRESS_NONE=y

Hmm, I thought I had explicitly forced CONFIG_MODULE_COMPRESS_NONE=y here.
How do you encounter CONFIG_MODULE_COMPRESS_XZ=y ?
In similar vein, debug was forced disabled here.

How are you getting around these? (I suppose, "by editing config after those changes are done").

This is not necessary anymore since armùbian-next rework removed this code.

Indeed. After armbian-next, we work on a copy of the build dir, and only for packaging, not for building -- that's half the reason we've a tmpfs, so we can clean/work/massage the built tree for packaging, without throwing away the build cache itself.

Maybe also I do not see much difference because I am CPU constrained. You might have beast I cannot compete with.

Much to the opposite. Igor has some beasts, yes, but I myself build on more constrained devices, even on SD/eMMC cards, and I/O slowness is as much a factor as CPU-limitedness.... :-D

---

Either way this is very interesting and supporting debug builds are something we should revisit for 23.08. Thanks for the investigations

[prahal]

Hmm, I thought I had explicitly forced CONFIG_MODULE_COMPRESS_NONE=y here.
How do you encounter CONFIG_MODULE_COMPRESS_XZ=y ?
In a similar vein, debug was forced disabled here.

I have a copy of the armbian kernel config from back then with my two changes (compression and debug). I copied it into userpatches/linux-rockchip64-edge.config from back when I first build with debug info. Might be outdated.

Also, now that I doubled checked the -dbg package is not built by armbian-next build framework.
It was built correctly with the previous armbian framework with the two changes I told above (building without compression and moving the create debug directory in builddeb before the "make M=scripts clean" armbian added to its cross compile builddeb code.

This is not a matter of simply adding back the CONFIG_DEBUG_INFO=y back as I already have it in my config.
The code to detect CONFIG_DEBUG_INFO is not there neither the one to prepare the -dbg package.

The modules are still built with debug symbols thus the bigger build size. So maybe the debug symbols are still shipped but in the non -dbg package, this is as there is no symbol stripping command.
So debug builds might already work, only not shipped in /usr/lib/debug by a -dbg deb package.

Indeed. After armbian-next, we work on a copy of the build dir, and only for packaging, not for building -- that's half the reason we've a tmpfs, so we can clean/work/massage the built tree for packaging, without throwing away the build cache itself.
Note that in the case of a docker build this /armbian/.tmp/work- is cleaned either way when the container is destroyed. So the mounting workdir in a tmpfs is duplicating the auto clean.
Though I admit that with the tmpfs you are safe even without a docker build container.

Much to the opposite. Igor has some beasts, yes, but I myself build on more constrained devices, even on SD/eMMC cards, and I/O slowness is as much a factor as CPU-limitedness.... :-D
OK, that may be why on my SSD build box as the storage is fast, the copying in the workdir looks as fast on plain storage than on SSD (I mean the time is so short for both that it adds no value).

[rpardini]

Yes. Thanks for explanation.

I, in fact, failed to port over the -dbg .deb packaging from back when we had 17+ different mkdebian/builddeb patches. (also happened to libc-headers, but that's a different story).

Looks to me we need simple yet official way to, simultaneously

• enable CONFIG_DEBUG_INFO (and maybe dwarf stuff, etc)
• disable usage of tmpfs for packaging
• bump up the size of /boot, if separate partition
• account for -dbg package, if needed...
• possibly have different package names ("-debug") to avoid conflicts

could be something as simple as KERNEL_DEBUG=yes, what do you think? And most importantly, would you be interested in leading this effort?

Note that in the case of a docker build this /armbian/.tmp/work- is cleaned either way when the container is destroyed.

Actually, in our Docker impl, .tmp is a Docker anonymous volume, which is not a tmpfs. It is usually backed by disk (/var/lib/docker), not RAM. It is isolated to that container, though, and thrown away after the run is done.
That way the tmpfs done inside the container is not wasteful.

(temporary) Logs are also handled the same tmpfs way -- that makes a huge difference on slow I/O machines.

I mean the time is so short for both that it adds no value

On fast I/O subsystem, indeed. A lot of us build on small SBCs with only eMMC/SD, believe it or not. (definitely not debug kernels, though).

[prahal]

could be something as simple as KERNEL_DEBUG=yes, what do you think? And most importantly, would you be interested in leading this effort?

Not for now. But I definitely added already added it to my todo.

bump up the size of /boot, if separate partition

This might not be required. The debug symbols are stored in /usr/lib/debug

possibly have different package names ("-debug") to avoid conflicts

I would keep it only the linux-image debug symbol package with the -dbg suffix as upstream does, wouldn't you?

disable usage of tmpfs for packaging

if there is already tooling to detect the RAM size we could keep the tmpfs if 10G is available. It is just that the dpkg-deb step duplicates the whole kernel. 8G is not enough though.
What I do not know is if the workdir can contain more than just a package file at a time. Because I only build the kernel or u-boot. Though I doubt that the whole image content is expected to fit on the tmpfs.

account for -dbg package, if needed...

What do you mean by "account for"

enable CONFIG_DEBUG_INFO (and maybe dwarf stuff, etc)

I noted that PAHOLE config needs to be set, though the correct values are above my league. I simply refreshed the config with pahole installed. Without that, the build fails.
Would the CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT be best, or is it fine to hardcode CONFIG_DEBUG_INFO_DWARF5=y?

[prahal]

A few items I noticed, I still cannot work on adding the -dbg support though.

bump up the size of /boot, if separate partition
I believe this is not an issue as the debug symbols in the -dbg packages are stripped and shipped in /usr/lib/debug/ so /boot should not get bigger.

I have an issue that triggers at least when I build with the debug symbols but not stripped from the kernel image and modules (ie not in a -dbg package), that is the image and initramfs get too big for default armbian u-boot offset settings and thus u-boot fails to load the kernel and initramfs with invalid image errors. It could be that having these symbols splt in /usr/lib/debug -dbg would avoid this issue.

Maybe this bug report should be renamed to encompass a request to implement -dbg package generation.