Guide on running a Raspberry Pi with a ZFS root filesystem (Ubuntu/Manjaro ARM)
This guide is heavily inspired and derived from the work done by the zfsonlinux folks. I'm sharing my experience on May 12, 2020, having little success with building zfs on the 32 bit OS that is raspbian.
Note - On Ubuntu, the only downside of this approach as of writing is the Pi takes a few minutes to boot. It's getting caught up in some systemd service that eventually times out.
A friend suggested using the latest 20.04 Ubuntu Raspberry Pi edition, since it is 64 bits and the main x86 branch has zfs support. I downloaded the Pi4 tar.xz here: https://ubuntu.com/download/raspberry-pi and loaded it onto a 32GB SD card. For Manjaro, I downloaded a copy here: https://manjaro.org/downloads/arm/raspberry-pi-4/arm8-raspberry-pi-4-kde-plasma/
NOTE: This Ubuntu install will (somewhere I can't find) invoke a resize2fs to fill your SD card after every boot. I circumvented this by making an additonal dummy ext4 partition that fills my SD card, leaving me with the standard fat32 /boot parition, and two ext4s, one blank and one with my standard ubuntu OS. I then took the card and plugged it into the Pi4. SSH was already enabled, so I could do ssh ubuntu@ubuntu, password ubuntu. You might have to find the IP of the host if your router doesn't support reverse dns hostname lookup through DHCP. Manjaro also fills up the SD card. For it, I let it do its thing with the resizing, then shrinked the partition on another computer, but I'm sure both methods work for each.
Next, I kept my eye on this guide from the zfsonlinux github wiki: https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-install-Raspbian-to-a-Native-ZFS-Root-Filesystem,-or,-How-I-Learned-to-Love-Data-Integrity. It goes through how to do this on Raspbian, but I hit little luck with this. (I compiled zfs right out of github and copying my root fs was hanging somewhere in the internals of ZFS [kernelspace] - not fun).
As most projects go, I did some googling to start for Ubuntu, and found this reddit post: https://www.reddit.com/r/zfs/comments/ekl4e1/ubuntu_with_zfs_on_raspberry_pi_4/
A commenter suggests running sudo apt install zfs-dkms. So I did. It seemed to successfully install zfs (I could do zfs and zpool commands), but I took it one step further and install two additional packages after this completed building - zfsutils-linux and zfs-initramfs (installing these didn't hurt, and we likely need something from zfs-initramfs for this project).
I skipped all the building parts / paritioning of the above wiki tutorial, since we just did with two apt get commands (the hard part is done)
For Manjaro, I used the Arch Wiki: https://wiki.archlinux.org/index.php/ZFS, installed yay for the AUR, and did yay -S zfs-dkms, which compiled and installed zfs (with the only issue being GPL/CDDL, fix here: openzfs/zfs#9948 (comment), you'll need to flip the license to GPL (if this is a hobby install) and rerun dkms with sudo dkms install --no-depmod -m zfs -v 0.8.4 -k 5.4.51-2-MANJARO-ARM. yay might also fail on gpg keys, which if so, I imported mine like this:
gpg --keyserver pool.sks-keyservers.net --recv-keys C33DF142657ED1F7C328A2960AB9E991C6AF658B
gpg --keyserver pool.sks-keyservers.net --recv-keys 4F3BA9AB6D1F8D683DC2DFB56AD860EED4598027
For Manjaro, I enabled the systemctl configs as suggested and added the zfs module to modprobe (see instructions in the wiki). Then you'll need to generate the initramfs with the zfs module inside. I did this by modifying /etc/mkinitcpio.conf, adding zfs before filesystems in the array (see https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS#Install_and_configure_Arch_Linux), then running mkinitcpio -P to regenerate the initramfs.
We're actually ready to make our pool on top of the dummy 3rd parition we made (make sure it looks right in parted)
You should be able to run this (as recommended in the zfsonlinux wiki)
sudo zpool create -o ashift=12 -O canmount=off -O compression=lz4 -O normalization=formD -O mountpoint=/mnt/rastank -O xattr=sa -O relatime=on -O recordsize=1M -R /mnt/rastank rastank /dev/mmcblk0p3
sudo zfs create -o mountpoint=/ rastank/ubuntu
^^^ you'll need to put in -f somewhere in the zpool command to force overwriting that dummy partition. I'll let you put that in because the above command is indeed destructive with -f. Make sure you're on the right Pi, and mmcblk0p3 is really the extra parition.
The second command is creating our root dataset, hence we tell it to mount at /.
If you want to segregate out different pieces of your filesystem, this is the time (make /var, /home, etc). This is how I did mine (with zfs create -o mountpoint=<> rastank/ubuntu/<>)
rastank 2.15G 25.9G 96K /mnt/rastank
rastank/ubuntu 2.14G 25.9G 1.09G /
rastank/ubuntu/home 876K 25.9G 164K /home
rastank/ubuntu/home/lancache 360K 25.9G 360K /home/ubuntu/lancache
rastank/ubuntu/var 1.02G 25.9G 76.7M /var
rastank/ubuntu/var/lib 812M 25.9G 295M /var/lib
rastank/ubuntu/var/lib/docker 507M 25.9G 92.0M /var/lib/docker
rastank/ubuntu/var/log 17.7M 25.9G 2.69M /var/log
rastank/ubuntu/var/spool 232K 25.9G 120K /var/spool
(Replace Ubuntu with Manjaro or whatever you prefer)
If you use docker, having /var/lib/docker as a zfs filesystem (probably with snapshots off) will default it to using the zfs driver for managing volumes.
Continuing with the guide, I ran this apt install: sudo apt-get install pv di dialog to get some insight into the next tar copy. (Manjaro just needs pacman -S pv
We're ready to copy our filesystem off of the ext4 parition onto our zfs dataset, which should be mounted at /mnt/rastank. Run this to copy it:
(cd /; sudo tar cf - --one-file-system . ) | pv -p -bs $( sudo du -sxm --apparent-size / | cut -f1 )m | (sudo tar -xp -C /mnt/rastank)
You might get an error about snapcraft on Ubuntu, just ignore it. I haven't tested snapcraft but I'm sure a reinstall would do the trick.
Remove the fstab entry for the ext4 / partition (it actually works with this in but it's good practice): vi /mnt/rastank/etc/fstab
Make sure zfs is in our initramfs (should be from our apt install earlier): lsinitramfs /boot/initrd.img-5.4.0-1008-raspi | grep bin/z. For Manjaro, I checked with strings *img | grep zfs in the initramfs img in /boot.
I added this in /boot/firmware/usercfg.txt only for Ubuntu. Not sure if it's required/matters:
initramfs initrd.img-5.4.0-1008-raspi followkernel
Your kernel might be different than mine. Do uname -r and replace (or just try without this and make a PR if it works!)
Next we need to tell the initramfs what to do. This is done in cmdline.txt. Here's mine for Ubuntu:
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ZFS=rastank/ubuntu rootfstype=zfs elevator=noop rootwait fixrtc plymouth.enable=0 zfs.zfs_arc_max=67108864
...and here's mine for Manjaro:
zfs=rastank/manjaro rootfstype=zfs rw rootwait console=ttyAMA0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyAMA0,115200 zfs.zfs_arc_max=67108864 elevator=noop snd-bcm2835.enable_compat_alsa=0
You can copy this over top of your existing cmdline.txt. We're telling initramfs that our root volume is of type zfs and is the dataset rastank/ubuntu (or whatever you called it)
reboot. On Ubuntu, after a few minutes of it complaining about a missing file in /tmp, which has few hits on Google, it should continue the boot process and give a prompt + ssh access. If you figure this out please open an issue or PR this guide. Manjaro starts without any noticiable hickups.
If you do df -h you should see your new rootfs as the zfs dataset name:
ubuntu@ubuntu:/boot/firmware$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 380M 3.9M 376M 2% /run
rastank/ubuntu 28G 1.4G 27G 6% / <--------------------------------
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/loop0 49M 49M 0 100% /snap/core18/1708
/dev/loop1 62M 62M 0 100% /snap/lxd/14808
/dev/loop4 49M 49M 0 100% /snap/core18/1753
/dev/loop2 62M 62M 0 100% /snap/lxd/14958
/dev/loop3 24M 24M 0 100% /snap/snapd/7267
/dev/mmcblk0p1 253M 101M 152M 40% /boot/firmware
tmpfs 380M 0 380M 0% /run/user/1000
ubuntu@ubuntu:/boot/firmware$
Awesome! You're running a zfs rooted raspberry pi (high five!). There's one more optimization I made before calling it a successful night. We still have that old ext4 partition with our old root. I went ahead and added that partition in the pool by doing this:
zpool add rastank /dev/mmcblk0p2
Again, this needs -f to work - make sure mmcblk0p2 is the ext4 partition that was our old root fs and you're on the proper Pi.
Cool! Now you have 3 partitions, one for loading the initramfs (that fat32 parition) and two zfs partitions.
Do a zpool status to see your pool:
root@ubuntu:~# zpool status
pool: rastank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rastank ONLINE 0 0 0
mmcblk0p3 ONLINE 0 0 0
mmcblk0p2 ONLINE 0 0 0
errors: No known data errors
root@ubuntu:~#
And zpool list to see how much space you have:
root@ubuntu:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rastank 28.8G 1.40G 27.3G - - 0% 4% 1.00x ONLINE -
root@ubuntu:~#
Awesome! We turned on lz4 compression in our zpool create, so check on your compression ratio with a fresh install:
root@ubuntu:~# zfs get compressratio,used,logicalused /
NAME PROPERTY VALUE SOURCE
rastank/ubuntu compressratio 1.80x -
rastank/ubuntu used 1.40G -
rastank/ubuntu logicalused 2.30G -
root@ubuntu:~#
Check that out! We're saving 1.1Gigs of SD card storage with lz4 compression. It's very low overhead and a Pi4 should be able to handle it.
If you're wondering where some of your storage went (28.8G in zpool list vs 26.5G in zfs list) - read up on spa_slop_shift. TLDR zfs is reserving some space for things like snapshot, so you can still manage the pool on a full filesystem: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSSpaceReportDifference
You can modify the value by echoing into this /sys/ file: /sys/module/zfs/parameters/spa_slop_shift if you want to reclaim some space, but just know the risks you are taking. See this question I asked the openzfs community for details on what the number means: openzfs/zfs#10260 (comment)
That's it! Feel free to do whatever you want with this Pi! It's a working Linux system with ZFS storage underneath.
I personally like turning on snapshot visibility with zpool set listsnapshots=on rastank.
If this article helped you, please leave a comment in the open issue and let me know! I'd love to hear your use cases/feedback.