[ENHANCEMENT] Migrating jails between Bastille servers

[VulcanRidr]

I am in the process of migrating off of iocage, bastille is awesome so far. One of the very cool features is that you can "hot" export jails.

I have two mini-servers (Lenovo Thinkcentre m710qs) that are my jail servers. A really nice feature would be for some type of telemetry between the bastille processes to talk to each other, and be able to hot swap jails "on the fly".

Initially, it would be nice to extend bastille export functionality you could do something like "bastille export --host ...Then through the telemetry link, shut down the jail on the old host.

I wrote a set of scripts for iocage where I could iocage export and then copy the files over to the other host, and shut the jail down on the first host that I would be happy to share.

[SnapshotCiTy]

@VulcanRidr I'd be thankful to get a look at your iocage migration scripts if you'd accept to share.

About Bastille migration of jails between hosts, I stumbled upon this article, let us know if that works for you and solves it. But yes, I upvote that - in-the-tool migration of jails between hosts would be great, even for the purpose of getting a host ready to take over in case of failure of the first one.

https://hackacad.net/post/2023-04-28-migrating-multiple-bastille-jails-new-host/

Slightly out of topic: I'm looking for info/ressources to be more HA with jails - if you have any info, articles, I'm very interested. Would be great to do HA jails with Bastille.

[VulcanRidr]

Sure! You may get them from https://github.com/VulcanRidr/iocage_jail_mgmt. Be warned they work, but they are really ugly and kludgey.

I will take a look at that hackacad article.

Now that I have things more fully stood up, I can give you some details on my setup, and other thoughts on how I am using it...I have two jail hosts, erebus and equinox, running Bastille. I have mounted the backups directory on both to my TrueNAS box, so all backups are shared.

I am converting all of my jails from thin to thick jails, since they are long-term servers, and since I am having problems upgrading jails from 13.2 to 14.0. So I usually run all of the jails on one host or the other, but especially while I am converting the jails, I have all of the 13.2 ones on equinox, and then recreating them on 14.0 on erebus. But I can see other reasons why I would need to split jails across hosts.

So these are some of my thoughts. I know you can use the bastille_list directive to tell bastille which jails to start, but what about

• An option in bastille to associate one or more bastille servers, and then set flags for each jail servers as to which jails are "primary" on that jail.
• A check whenever starting a jail to see if it is already running elsewhere.
• a way to do something like "bastille move "
• You could also stop a jail on one server and start it manually (or create it) on a different one, as I am doing with my jails to convert to thick and 14.0.

[VulcanRidr]

You know, as I was mulling this over, another thought occurred to me. Since, unlike iocage, bastille takes advantage of thesnapshot feature of ZFS, instead of going down the route of exporting and importing jails, you could easily migrate jails by doing a zfs send /usr/bastille/ and any associated datasets (root, pgsql, mysql, etc). I always create separate pgsql and mysql datasets, since the ideal recordsizes are 8k and 16k respectively, though it appears I am going to have to fix my database datasets for the jails since they are set at 1M...Sigh.

In any case, I believe a mechanism that allows you to migrate jails through the zfs send | zfs recv would be faster than exporting and importing...

[SnapshotCiTy]

Thanks for the info you provide - this puts me on the trail to try to setup bastille (i want to move from Iocage to Bastille) and to set-up the migration or quick switch-over (in case of the main host going offline) of bastille-managed-jails from one host to another - through zfs send/rcv...

[VulcanRidr]

Cedric, just for the record, I have changed the way that I have my backups set up. I have started NFS mounting my /usr/local/bastille/backups from my NAS box on both jail servers. Then I run something similar to the following to back up the jails:

bastille list | grep -v Hostname | awk '{ print $3 }' | while read x ; do bastille export --xz $x ; done

which makes the jail backups immediately available on both servers.

[VulcanRidr]

Cedric, just for the record, I have changed the way that I have my backups set up. I have started NFS mounting my /usr/local/bastille/backups from my NAS box on both jail servers. Then I run something similar to the following to back up the jails:

bastille list | grep -v Hostname | awk '{ print $3 }' | while read x ; do bastille export --xz $x ; done

which makes the jail backups immediately available on both servers. I could write a wrapper script around this, but am considering the possibility/feasability of doing jail management through ansible.