How do I add a second bootable disk?

[jimsalterjrs]

I've got a working ZFSBootMenu system set up, but I cannot for the life of me figure out how to get multiple drives bootable.

After installing the system to a single disk, I first tried simply copying the partition table from sda to sdb, then copying the files from sda1 (/boot) to sda2. No joy--attempting to copy the files directly fails with REALLY strange errors up to and including "cannot allocate memory." So did workarounds like tarring and untarring or zipping and unzipping. So I gave up on that.

Next thing I did was just follow the install instructions for the second drive, but skipping the parts about the pool (which is, after all, already installed) and changing /boot to /boot2. That worked in the sense of getting all the files there and creating EFI boot entries for them, but booting from sdb results in the dreaded "insert bootable media" message from my UEFI firmware.

What should the procedure look like? I install a LOT of multiple-drive systems, and I'd like to upgrade to ZBM--but I need to get issues with multiple-BOOT-drive setup fixed first. Thank you!

[ahesford]

I assume your reference to /dev/sda2 was supposed to be /dev/sdb1, i.e., the first partition on the second disk. If you've made /dev/sdb1 a proper EFI system partition and copied the files over, I suspect the problem is that telling your system to boot from the second disk is actually telling it to look at the fallback EFI executable on its EFI system partition. The fallback executable is located at \EFI\BOOT\BOOTX64.EFI; you can always copy the ZFSBootMenu EFI executable to this location and things should generally just work. If you want to mimic the same file layout as the first partition, you'll need to make sure that your firmware is actually configured to boot the specific file on the second disk. This can often be accomplished from firmware setup interfaces---both my ASUS motherboard and Dell laptops provide a method for choosing arbitrary EFI executables files on any attached disk.

If you want to keep things synchronized between the two, you have a few options. I've always favored Linux mdraid arrays for this task. I create an mdraid mirror with all of the EFI system partitions I'd like to replicated, mother make a common filesystem on the RAID device and populate accordingly. To make this work, the mdraid mirror must be created with --metadata=0.9 or --metadata=1.0 flags. These metadata versions put the mdraid header at the end of every partition, so the firmware will see each constituent partition as a standalone filesystem with identical contents. Newer metadata versions put the header at the beginning of the partition, so the firmware will lot recognize any filesystems on the devices.

Some people recoil at the use of mdraid in this way, because there is a small but nonzero chance that the firmware will write to whatever filesystem it reads during boot. This can corrupt the filesystem. However, the chance of any issues is incredibly small, and the EFI system partition is trivial to recover in the rare case it actually happens. (You can always boot a system from ZBM on a USB thumb drive if you need to recover from a bad EFI system partition.) If you are concerned about this possibility, you will have to treat every disk's EFI system partition as an independent filesystem and manually sync them. An example hook for generate-zbm shows how this can be accomplished automatically whenever a new ZBM image is created, or you can replicate the steps on demand.

[jimsalterjrs]

well, that unfortunately didn't fix it. I suspect a large part of the problem is this isn't a proper server board; I'm using a GaMeRRiFFiC!!!1 board with an IMO unusually stupid and buggy AMI BIOS.

I tried the steps you recommended for a portable boot (but using /dev/sda1 and /dev/sdb1, not using a thumb drive) and now neither partition will boot--both do the "insert media" dance. That's after manually running efibootmgr to point to sda and sdb--this idiot motherboard initially refused to even see either drive as bootable, until I manually forced it to see them with efibootmgr.

I guess the next thing is to try creating a ZBM USB recovery drive, because I am batting .000 on doing it directly on the drives--on this motherboard, at least. :(

[jimsalterjrs]

Good news is, I had no problems creating the portable drive (although I would certainly recommend filling out that guide a little more thoroughly, to avoid having to back-and-forth between it and pages that give example sgdisk and mkfs commands). So at least I know I can, in fact, recover the system.

I guess I'm going to have to give up on making individual drives bootable on their own right, though (on this system, anyway) and just use mdraid1 for the boot on both of them. Hopefully I can manage to get that working without needing to reinstall the whole system, but it shouldn't be a huge problem either way, since I can just replicate the entire system back onto the newly created pool if I have to go that nuclear on it.

[ahesford]

Can you share the output of the following commands?

1. efibootmgr
2. ls -R /mnt/sda1, assuming your /dev/sda1 is mounted there
3. ls -R /mnt/sdb1, assuming your /dev/sdb1 is mounted there

[jimsalterjrs]

Not anymore, because I finally got the accursed thing working properly using mdraid. That actually got this motherboard recognizing both drives AS bootable, which it was refusing to do at least half the time, after I used efibootmgr to manually create entries for both drives.

This was what worked, after booting into the system from a portable ZBM thumbdrive.

First, unmount /boot/efi and wipe the boot partitions on both disks:

root@new:~# umount /boot/efi
root@new:~# sgdisk --zap-all /dev/sda1 ; sgdisk --zap-all /dev/sdb1

Then, reconfigure both boot partitions to the Linux RAID automount type, create a new RAID1 mirror across them, and format it for use:

root@new:~# sgdisk /dev/sda -t1:FDOO ; sgdisk /dev/sdb -t1:FD00
root@new:~# mdadm --create --metadata=1.0 /dev/md0 -l1 -n2 /dev/sda1 /dev/sdb1
root@new:~# mkfs.vfat -F32 /dev/md0

Make sure that the new array will automount as /boot:

root@new:~# cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/md0 /boot/efi vfat defaults 0 0

Now, mount the array and drop the EFI executables onto it in the right place:

root@new:~# mount /boot/efi
root@new:~# mkdir -p /boot/efi/EFI/BOOT
root@new:~# wget -o /boot/efi/EFI/BOOT/BOOTX64.EFI https://get.zfsbootmenu.org/efi

And finally, use efibootmgr to inform my accursedly stupid motherboard that it should look for EFI boot options on each drive:

root@new:~# efibootmgr -c -d /dev/sda -p1 -L "ZFSBootMenu" -l '\EFI\BOOT\BOOTX64.EFI'
root@new:~# efibootmgr -c -d /dev/sdb -p1 -L "ZFSBootMenu (recovery boot disk)" -l '\EFI\BOOT\BOOTX64.EFI'

After doing all and I do mean all of the above, my idiot motherboard finally recognized both drives as bootable AND is capable of successfully booting from either of the internal drives or from the portable ZBM boot drive y'all helped me create earlier. Thank you.

For the record, I do not think most motherboards will encounter anywhere near this level of difficulty. But with this one... whew. This AMI BIOS is so stupid that it starts with like six different boot options, which it offers you the option to disable... only thing is, if you disable them, you will NEVER be able to recreate them. So now instead of being able to select six different boot options in my BIOS and order them as I like, I have one. ONLY one. There is literally no way to recreate new options. I can go into the BBS priorities to change which ONE disk my ONE boot option will boot from... but I cannot bring back the missing options AS options, if you follow my drift.

I really hate this motherboard, and I really hope this direct step-by-step all on one page way to get the damn thing working helps someone else at some point. :)