This will be very useful. I don't think it has to be complicated though with bootstrap overlays. All we need - some minimal os that boots first and then can properly load big OS container, setting up disks, grab secrets, etc. This is exactly how it is done in Brigth Cluster Manager as far as I remember and it worked perfectly well. No issues with old BIOS/pxe/whatever, also - easy stateless setup, but with disks. Meaning - every time node boots, disks are wiped out and OS just rsynced to disks.

This will be very useful. I don't think it has to be complicated though with bootstrap overlays. All we need - some minimal os that boots first and then can properly load big OS container, setting up disks, grab secrets, etc.

Treating the bootstrap exactly the way a normal container is treated keeps things consistent. Allowing the bootstrap to have overlays brings that same flexibility to the bootstrap phase and is effectively already written in WW4.

Rather than having a single bootstrap which is hard-coded, a site might want to build their own with dracut directly from a container, creating a container-specific bootstrap that does not need a kexec() call, just a switch_root to the final init and /. I think ( but haven't proven yet) that it'd be possible to produce a container-specific dracut built initrd that used systemd as the native init rather than needing to shim an init script in the boot process. For these and other reasons, the most important being "all the things I haven't thought of but someone else will", I think it's worth making the bootstrap process as configurable as the current container/overlay system to give admins that flexibility.

A follow up to this RFE would definitely be that WW should include a standard and documented bootstrap container that gives new users a working default, but given that it's just as simple to allow complete freedom for local sites to build their own bootstraps and processes, it'd be a shame to restrict to a single "approved" bootstrap image/process.

Also I want WW to be BETTER than Bright, not LIKE Bright 😁

I'm a supporter of this idea. If you came from WW3 (or even WW2/Perceus) you know that has been the model for years (2-stage provisioning). I believe we had some discussion on this during the early days of WW4 project and @gmkurtzer decided to go to current model. From what I recall the original argument was like, if you are already provisioning a container to the node, why not just provision it directly, instead of relying on another stager/bootstrap to provision it for you. But I do see the value of being flexible. Technically this is not too difficult to do with current model as well and you can build a 1st stage container that works just like the old bootstrap, and perform the necessary tasks such as preparing disks that @msquantori would like to do, then wget the second stage container from the same warewulfd, extract it, and switch_root to it.

My use case for the 2-stage provisioning is different than yours. So just throw it out as well for some thoughts. We have some hardware (NIC in this case) that lack of ipxe support, basically after some fiddling with ipxe itself I was only able to get it negotiate to 100 Mbps instead of the expected 1 Gbps (I think there are similar reports in ipxe community as well). This causes downloading the full-blown container image taking a lot of time. However the OS driver does not have that limit so the use case would be to just use the ipxe to download the 1st stage container as small as possible, then load up the OS driver and download the 2nd stage full-blown container. Yes I agree that a native support would be ideal but just saying that this isn't a showstopper with the current model.