Emergency shell on boot after kernel upgrade (crc32c module not loaded)

[Jcd1230]

Hi,
Today I decided to pacman -Syu my Arch system and reboot, and to my surprise I couldn't SSH and had to use the DigitalOcean console to find out it failed to load the filesystem due to missing the crc32c module, and it was dropped to an emergency shell.
The problem for me was fixed by adding crc32c to the MODULE=" ..." list in /etc/mkinitcpio.conf before running the system update.

Unfortunately, once this problem occurs It seems like your Droplet is completely hosed!
It seems there is no (good?) way to download snapshots/backups of droplets so that the boot issue can be fixed manually, so I had to restore to a snapshot I made just after running this script. While I was at it, I did test running the system upgrade from that snapshot, and the same problem existed there.

So, this problem might exist specifically on the system, the snapshot I made (22 days old, so it was made with the most up to date version of this script), or with the upgrade itself. I will test it again with a fresh Debian droplet tomorrow and run the script on it to see if it is a problem with a fresh system or not.

Luckily the Droplet that was affected for me was essentially a throwaway, hopefully this doesn't cause anyone any real grief!

[injust]

I just tested the script with a fresh system. Booting the droplet with a restored snapshot works without a hitch.
Did you run the script with any extra parameters?

[Jcd1230]

Nope, no custom parameters.
I just tested it again on a fresh Debian 9.1 64 bit droplet ($5 tier, Toronto server (TOR1))
Booting the droplet does work right after the script. Also, rebooting the droplet works fine in that case as well.
However, I ran mkinitcpio -p linux, and then tried rebooting and I got the same problem.
Since the mkinitcpio step is run whenever the linux package is upgraded, this problem will occur as soon as the kernel is updated.

Just a few quick googles reveals this is not a completely unique issue [1] [2]... seems to be that ext4 depends on the crc32c module, and somehow it is getting added into the initramfs when you build it the first time, but mkinitcpio.conf is left without it (or any other modules for that matter. I wonder if there any other modules that should be loaded as well that are getting lost?)

Anyways, see if you can reproduce after running mkinitcpio -p linux, if not I have no idea where the issue could be coming from.

Screenshot of the console after rebooting.

[1] https://lists.debian.org/debian-kernel/2016/04/msg00013.html
[2] https://bugs.archlinux.org/task/49380

[fcladera]

Hello,

I updated my main droplet this morning, and I discovered this issue when I got stuck in the emergency shell.

If you arrive at this point, and you have valuable data in your droplet (and old snapshots), you may get it back using the recovery kernel.

1. Power off your droplet using the DO web interface
2. Mount the Recovery Kernel
3. Start your droplet, launch the online VNC connection
4. Mount your root partition, chroot (https://wiki.archlinux.org/index.php/change_root)
5. Install an old kernel
6. Power Cycle using the DO web interface

---

Anyway, this is the first time in +2 years that I have an issue with my droplet. Thanks for the awesome work!

[gh2o]

That's interesting (and unfortunate 😢). Just ran pacman -Syu on mine and it booted fine... but then I realized that that was because I was using btrfs, which doesn't depend on crc32c.

Was the fallback initramfs (/boot/initramfs-linux-fallback.img) missing the crc32c module too?