Bootable Ubuntu Image is often stuck at loading initrd

[shady786]

mkosi commit the issue has been seen with

main

Used host distribution

Ubuntu Jammy Jellyfish

Used target distribution

Ubuntu Focal

Linux kernel version used

5.15.0-107-generic

CPU architectures issue was seen on

None

Unexpected behaviour you saw

The image is halting for an excess of 5 minutes at the load of initrd before progressing and eventually loading.

Used mkosi config

[Config]
MinimumVersion=22

[Distribution]
Distribution=ubuntu
Release=jammy
Repositories=main,restricted,universe,multiverse
Architecture=x86-64

[Output]
Format=disk
ManifestFormat=json
ImageId=test
ImageVersion=1
OutputDirectory=mkosi.output
CompressOutput=no

[Content]
Locale=en_US.UTF-8
Keymap=de
Timezone=Europe/Berlin
Bootable=yes
Bootloader=systemd-boot
WithNetwork=yes
Packages=linux-image-generic
         systemd
         systemd-sysv
         udev
         dbus
         linux-firmware
         intel-microcode
         apt
         apt-utils
         bash
         iproute2
         iptables
         net-tools
         netcat
         netplan.io
         curl
         efibootmgr
         gnupg
         ntp
         language-pack-de
         language-pack-de-base
         sudo
         unattended-upgrades
         i3
         xorg
         imagemagick
         conky-all
         qrencode
         apt-transport-https
         vim
         nano
         less
         openssh-server
         alsa-base
         pulseaudio
         ffmpeg
         vainfo
         libavcodec-extra
         libavcodec-extra58
         libvo-amrwbenc0

[Host]
ToolsTree=default

mkosi output

No response

[DaanDeMeyer]

Please provide more information. You give us hardly anything to work with, how exactly are you booting the image? What hardware or virtualization software are you using? Please give us something to work with. As it is this bug report is not actionable.

[shady786]

Hi, I have attempted to boot the raw image on utm using emulation for mac silicon and also tried to dd it to a flash drive and boot from this on a physical device. I can confirm images I have built previously using v14 are working fine on both.

The obvious upgrade from v14 to v22 (main) items I've ruled out as I do match the required packages for EFI boot as per your docs. The log is too large to add as a comment, but I include the systemd-boot log extract from the output below. Is there anything else I could provide which would be helpful to you ?

‣ Installing systemd-boot…
Created "/buildroot/efi/EFI".
Created "/buildroot/efi/EFI/systemd".
Created "/buildroot/efi/EFI/BOOT".
Created "/buildroot/efi/loader".
Created "/buildroot/boot/loader".
Created "/buildroot/boot/loader/entries".
Created "/buildroot/boot/EFI".
Created "/buildroot/boot/EFI/Linux".
Copied "/buildroot/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/buildroot/efi/EFI/systemd/systemd-bootx64.efi".
Copied "/buildroot/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/buildroot/efi/EFI/BOOT/BOOTX64.EFI".
Random seed file /buildroot/efi/loader/random-seed successfully written (32 bytes).
‣ Generating system users
‣ Generating volatile files
‣ Applying presets…
Failed to preset unit, unit /buildroot/etc/systemd/system/sudo.service is masked.
Failed to preset unit, unit /buildroot/etc/systemd/system/NetworkManager.service is masked.
Failed to preset unit, unit /buildroot/etc/systemd/system/gdm3.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/cryptdisks-early.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/cryptdisks.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/rcS.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/pulseaudio-enable-autospawn.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/x11-common.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/hwclock.service is masked.
Failed to preset unit, unit /buildroot/usr/lib/systemd/system/rc.service is masked.
Unit /buildroot/etc/systemd/system/sudo.service is masked, ignoring.
Unit /buildroot/etc/systemd/system/NetworkManager.service is masked, ignoring.
Unit /buildroot/etc/systemd/system/gdm3.service is masked, ignoring.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/oneshot-configure.service → /etc/systemd/system/oneshot-configure.service.
Unit /buildroot/usr/lib/systemd/system/cryptdisks-early.service is masked, ignoring.
Unit /buildroot/usr/lib/systemd/system/cryptdisks.service is masked, ignoring.
Unit /buildroot/usr/lib/systemd/system/rcS.service is masked, ignoring.
Created symlink /buildroot/etc/systemd/system/sysinit.target.wants/systemd-sysext.service → /usr/lib/systemd/system/systemd-sysext.service.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/metricbeat.service → /usr/lib/systemd/system/metricbeat.service.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/salt-minion.service → /usr/lib/systemd/system/salt-minion.service.
Unit /buildroot/usr/lib/systemd/system/pulseaudio-enable-autospawn.service is masked, ignoring.
Created symlink /buildroot/etc/systemd/system/ctrl-alt-del.target → /usr/lib/systemd/system/reboot.target.
Unit /buildroot/usr/lib/systemd/system/x11-common.service is masked, ignoring.
Unit /buildroot/usr/lib/systemd/system/hwclock.service is masked, ignoring.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/remote-cryptsetup.target → /usr/lib/systemd/system/remote-cryptsetup.target.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/filebeat.service → /usr/lib/systemd/system/filebeat.service.
Created symlink /buildroot/etc/systemd/system/multi-user.target.wants/remote-veritysetup.target → /usr/lib/systemd/system/remote-veritysetup.target.
Created symlink /buildroot/etc/systemd/system/sockets.target.wants/ssh.socket → /usr/lib/systemd/system/ssh.socket.
Unit /buildroot/usr/lib/systemd/system/rc.service is masked, ignoring.
Created symlink /buildroot/etc/systemd/user/timers.target.wants/systemd-tmpfiles-clean.timer → /usr/lib/systemd/user/systemd-tmpfiles-clean.timer.
Created symlink /buildroot/etc/systemd/user/sockets.target.wants/pulseaudio.socket → /usr/lib/systemd/user/pulseaudio.socket.
Created symlink /buildroot/etc/systemd/user/default.target.wants/pulseaudio.service → /usr/lib/systemd/user/pulseaudio.service.
Created symlink /buildroot/etc/systemd/user/basic.target.wants/systemd-tmpfiles-setup.service → /usr/lib/systemd/user/systemd-tmpfiles-setup.service.
‣ Applying first boot settings
/buildroot: /etc/locale.conf written.
/buildroot: /etc/vconsole.conf written.
/buildroot: /etc/localtime written
‣ Generating hardware database
‣ Recording packages in manifest…
‣ Generating disk image
Pre-populating ext4 filesystem of partition 10-root.conf twice to calculate minimal partition size
Populating ext4 filesystem.
Successfully populated ext4 filesystem.
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: 0/268435456������������������� �������������������done
Creating filesystem with 268435456 4k blocks and 67108864 inodes
Filesystem UUID: 1cfdce02-9cd9-42d2-b216-360c024076fb
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848

Allocating group tables: 0/8192��������� ���������done
Writing inode tables: 0/8192��������� ���������done
Creating journal (262144 blocks): done
Copying files into the device: done
Writing superblocks and filesystem accounting information: 0/8192��������� ���������done

/var/tmp/.#repart88fe9b9855d7c334 successfully formatted as ext4 (label "root-x86-64", uuid 1cfdce02-9cd9-42d2-b216-360c024076fb)
Minimal partition size of ext4 filesystem of partition 10-root.conf is 6.4G
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: 0/1689162��������������� ���������������done
Creating filesystem with 1689162 4k blocks and 422656 inodes
Filesystem UUID: d853d13a-3bd4-4a7c-be97-4df9ad725a48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: 0/52����� �����done
Writing inode tables: 0/52����� �����done
Creating journal (16384 blocks): done
Copying files into the device: done
Writing superblocks and filesystem accounting information: 0/52����� �����done

/var/tmp/.#repart88fe9b9855d7c334 successfully formatted as ext4 (label "root-x86-64", uuid d853d13a-3bd4-4a7c-be97-4df9ad725a48)
Automatically determined minimal disk image size as 6.9G.
Sized '/home/runner/.cache/mkosi/mkosi-workspacecu27u98n/staging/development_20240516.6330382.raw' to 6.9G.
Applying changes to /home/runner/.cache/mkosi/mkosi-workspacecu27u98n/staging/development_20240516.6330382.raw.
Copying in '/var/tmp/.#repart88fe9b9855d7c334' (6.4G) on block level into future partition 1.
Copying in of '/var/tmp/.#repart88fe9b9855d7c334' on block level completed.
Block level copying and synchronization of partition 1 complete in 1min 35.374482s (69.1M/s).
Adding new partition 1 to partition table.
Writing new partition table.
All done.

[DaanDeMeyer]

In v14 we still built the initrd using dracut, we've since switched to building the initrd with mkosi itself. So your issue is likely caused by the mkosi initrd including all kernel modules and firmware that is installed in the image. We have to include all modules and firmware because we do not know where the image will be installed. Search the documentation for "KernelModules" to find the settings that allow you to exclude specific modules (and any firmware they require) from the initrd built by mkosi.

[shady786]

This makes sense, I will look into the modules and report back.

[shady786]

Hi, I have some new information. When looking at the generated ESP Partition I notice that the loader.conf is actually empty and no entries exist in the entries dir.

loader.conf
#timeout 3
#console-mode keep

[DaanDeMeyer]

Hi, I have some new information. When looking at the generated ESP Partition I notice that the loader.conf is actually empty and no entries exist in the entries dir.

loader.conf #timeout 3 #console-mode keep

That's because we build unified kernel images. Those will be in <esp-mountpoint>/EFI/Linux

[shady786]

This is not actually a issue or bug. I needed to match all the required kernel modules and firmware.