How does the OSTree boot process work?

[fwilhe]

Hi,

this question is based on a project I'm doing. I'm tying to get a version of GardenLinux running with OSTree.

GardenLinux is a derivative of Debian Testing, and since it is focussed on running Gardener/k8s nodes I think that OSTree is a great fit. We do diverge from a default Debian install in some areas, for example we don't include GRUB in the distribution but use systemd-boot instead, and we use dracut for the initramfs generation. We're not (yet) using UKI, but that might be a thing in the future.

But I do struggle to get a PoC running, and I have some questions about the boot process.

I believe my questions are not answered by the documentation, but it might also be possible that I don't fully understand the docs.

We don't use composefs, and we don't run ostree-prepare-root as pid1.

I'll try to summarise what my current understanding is:

• The ostree dracut module puts ostree-prepare-root.service and ostree-prepare-root into the initramfs
• ostree-prepare-root is being run during the boot, and it's task is to setup the filesystem/mounts in a way so systemd can make switch-root just like in any other system
  • I've looked into it's code and I think I somewhat understood what's going on, but my confidence about this is not very high.
  • This will perform the following re-mounts:

• systemd's initrd-switch-root.service is running and calling systemctl --no-block switch-root /sysroot, the latter argument seems to be required in ostree systems
  • This will essentially mount /sysroot as /, leaving the former /sysroot/sysroot dir as /sysroot in the booted system.

My question is:

What are the specifics/expectations that OSTree has to get a bootable system?

I currently face issues with systemd complaining in switch-root that it is not able to find/chase it's own binary, even though I'm sure it is part of the root-fs that's in the OSTree commit:

Failed to chase our own binary /sysroot//lib/systemd/systemd: No such file or directory

I have read the deployment docs, and I do think my system does fulfil the mentioned requirements.

Can someone help me to understand how OSTree, systemd, dracut are working together and how that is different from a systemd & dracut system that does not use OSTree?

[cgwalters]

We'd love to have systemd-boot support, there's #1951 which it's blocked on.

even though I'm sure it is part of the root-fs that's in the OSTree commit:

Are you sure? Can you show the output of ostree ls $commit? In general what does your filesystem tree look like? Failing that, dracut usually has an emergency shell and that will let you e.g. ls /sysroot and look around.

The //lib looks a bit odd...I wonder if you're missing some toplevel symlink like /lib -> /usr/lib?

[fwilhe]

Hi @cgwalters, thanks for the hint, creating a symlink from /lib to /usr/lib seems to make a big difference. I get a lot of error messages while booting and a non-fuctional system, but at least I do get a shell now, that's new and that's great.

Are such requirements documented at some place, or would you consider this out of ostree scope? I've checked https://ostreedev.github.io/ostree/adapting-existing/.

I can share an ostree ls -R output, but I'm not sure if that now is still relevant.

Are you aware of any working debian based ostree systems? I know about Endlessos, but trying to run the builder does not work for me, it seems like it's not maintained.

[cgwalters]

That symlink is something that should be created at build time, while long ago I did contribute to Debian I don't know what should be creating it today.

Basically to start, you want what is here:

$ podman run --rm -ti debian:latest
root@2a513dc32631:/# ls -al /
total 24
dr-xr-xr-x.   1 root   root     12 Aug 17 13:42 .
dr-xr-xr-x.   1 root   root     12 Aug 17 13:42 ..
lrwxrwxrwx.   1 root   root      7 Jun 12 00:00 bin -> usr/bin
drwxr-xr-x.   1 root   root      0 Mar  2 13:55 boot
drwxr-xr-x.   5 root   root    360 Aug 17 13:42 dev
drwxr-xr-x.   1 root   root     18 Aug 17 13:42 etc
drwxr-xr-x.   1 root   root      0 Mar  2 13:55 home
lrwxrwxrwx.   1 root   root      7 Jun 12 00:00 lib -> usr/lib
lrwxrwxrwx.   1 root   root      9 Jun 12 00:00 lib32 -> usr/lib32
lrwxrwxrwx.   1 root   root      9 Jun 12 00:00 lib64 -> usr/lib64
lrwxrwxrwx.   1 root   root     10 Jun 12 00:00 libx32 -> usr/libx32
drwxr-xr-x.   1 root   root      0 Jun 12 00:00 media
drwxr-xr-x.   1 root   root      0 Jun 12 00:00 mnt
drwxr-xr-x.   1 root   root      0 Jun 12 00:00 opt
dr-xr-xr-x. 753 nobody nogroup   0 Aug 17 13:42 proc
drwx------.   1 root   root     30 Jun 12 00:00 root
drwxr-xr-x.   1 root   root     40 Aug 17 13:42 run
lrwxrwxrwx.   1 root   root      8 Jun 12 00:00 sbin -> usr/sbin
drwxr-xr-x.   1 root   root      0 Jun 12 00:00 srv
dr-xr-xr-x.  13 nobody nogroup   0 Aug 14 12:58 sys
drwxrwxrwt.   1 root   root      0 Jun 12 00:00 tmp
drwxr-xr-x.   1 root   root    116 Jun 12 00:00 usr
drwxr-xr-x.   1 root   root     90 Jun 12 00:00 var

right?

[fwilhe]

Hi @cgwalters thanks you helped me resolve the issue. I still get some systemd services failing because they don't expect certain directories to be read-only, but I have not yet investigated them in detail. Is there any specific documentation on how systemd and ostree work together or how systemd needs to be configured for ostree systems?