Pruning after template changes & sanoid performance

[danboid]

Up until now, we've been using sanoid 2.0.3 with this template:

[astarray/home]
	use_template = production
        recursive = zfs

#############################
# templates below this line #
#############################

[template_production]
        frequently = 4
        hourly = 24
        daily = 30
        weekly = 12
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

This Ubuntu 20.04 server has over 700 datasets and it seems that sanoid is bringing it to a crawl so I'm going to stop frequent snapshots in an effort to reduce the amount of effort sanoid has to put into creating and pruning snapshots. The pool consists of 6x 4 GB Samsung SATA 3 SSDs in a RAIDZ2 config.

If we've been running with frequent snapshots for over a year and stop creating frequent snapshots, will sanoids pruning script correctly destroy old frequent snapshots when its no longer creating new frequent snapshots or will I have to manually remove all of the old frequent snapshots?

Ubuntu 20.04 uses ZFS 0.8.3. I plan to upgrade this server to Ubuntu 22.04 soon which has ZFS 2.1.5. Does ZFS 2.1.5 handle large amounts of snapshots over hundreds of datasets better or am I not to expect to see any improvements for sanoid and snapshots after upgrading the pool and ZFS?

What tools or commands would you recommend I use to benchmark or monitor ZFS's performance when using different sanoid templates and/or different versions of ZFS?

The sanoid docs don't seem to cover recommendations for sensible template limits. I realise its impossible to cover all of the infinite different combinations of hardware and software that sanoid could be used on but some rough guidelines would be useful if you have any to offer?

I'm upgrading this server to sanoid 2.1.0 now so hopefully that will help a bit?

Thanks

[secabeen]

How often are you running sanoid? The default recomendation of '* * * * *' (every minute) is more often than needed for most setups, you may find much better performance only running every 5 or 15 minutes.

[danboid]

I'm just using the default timer settings of whatever the Ubuntu package uses so I presume it will be that. It seems like a good suggestion to me, I should be able to get away with only running sanoid once an hour instead of every minute yes.

Hopefully Jim will show up soon to answer the rest of my questions.

[danboid]

After just over a years use, this server ended up with almost 650K snapshots on its pool. I have run a server with a similar amount of users, configured to keep a similar amount of snapshots but using zfs-auto-snapshot. That has been running for over 4 years and it only has about 50K snapshots so it seems to me that sanoid's autoprune is not fit for purpose.

I have uninstalled sanoid and I am currently destroying all snapshots on the pool but it will take weeks to destroy all 650K snapshots and hopefully regain usable performance.

[jimsalterjrs]

Sanoid's autoprune does exactly what it's asked to do. You haven't shared your config, so I have no way of giving you any useful information on why you had more snapshots than you believed you should have.

For reference, this pool is nearly ten years old, and has been using Sanoid to manage its snapshots for the length of its existence:

jimbo@banshee:~$ zfs list -t snap | wc -l
3226

Best of luck with zfs-auto-snapshot.

[danboid]

This is the sanoid.conf that I used. I would liked to know if there's anything wrong with it.

[astarray/home]
	use_template = production
        recursive = zfs

#############################
# templates below this line #
#############################

[template_production]
        frequently = 4
        hourly = 24
        daily = 30
        weekly = 12
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

[danboid]

I was using the default version of sanoid in Ubuntu 20.04 for most of that year but I updated to the latest a few months ago. It didn't help clear up the excess snapshots. I updated the server to Ubuntu 22.04 a few months ago.

[jimsalterjrs]

There's nothing wrong with that, although recursive=zfs can surprise people when they don't immediately get snapshots of newly created datasets.

You should have 73 snapshots per dataset, possibly up to 78 snapshots per dataset relatively briefly if you catch it at the worst possible moment (when a new frequently, hourly, daily, weekly, and monthly has been created, before the next run which would delete the extra on all of the above).

Can you please provide the output of zfs list -r astarray/home | wc -l? Normally, I'd ask you for the output of zfs list -rt snap astarray/home | grep autosnap | wc -l as well, but with several hundred thousand snapshots on the pool, you'd likely be waiting for hours for that to complete.

Basically, the most likely issue seems to be server overload. If you get into a condition where multiple sanoid processes are stacking up because the system is overloaded, it's possible to wind up with a system that's too busy to ever manage to delete any old snapshots at all.

When you get into that condition, the solution is to limit how frequently you call sanoid from cron--on most systems, * * * * * is fine, because sanoid uses a cache file to avoid hammering zfs list -t snap too frequently. But on a really overloaded system, you can end up with multiple sanoid processes all trying to update an invalidated cache at once, and then you end up with the kinds of problems you're seeing.

Once you've limited how frequently sanoid is invoked, the next step is to clean up the old snapshots, either manually, or by directly invoking a single sanoid --cron --verbose from the command line, and letting it complete. Again, with as many extra snapshots as you're describing here, that could take quite some time... and you might want to consider commenting out your sanoid cron job entirely, until your manual cleanup (or manual CLI invocation of sanoid --cron --verbose) finishes.

[danboid]

Hi Jim

# zfs list -r astarray/home | wc -l
729

This server is using a 6 x 4GB SATA SSD RAIDZ2 array. I started destroying all snapshots in a for() loop 8 days ago and its still only destroyed 15% of them which means at the current rate it could take another 45 days to destroy them all! There's no faster way is there?

Whats odd about this is we have a second Ubuntu ZFS server that we use syncoid to mirror the snapshots onto. This is also using a 6 disk disk RAIDZ2 pool but its compromised of spinning rust disks. This machine had even more snapshots - it had over 800K snapshots in its pool but it took less than a week to destroy them all. Why might that be? Does ZFS really run that much slower on SSDs?

When I created the astarray zpool, I used a command like this:

zpool create -o ashift=13 -o autoexpand=on -o autoreplace=on -O atime=off -O compression=lz4 astarray raidz2 scsi-SATA_Samsung_SSD_870_S5STNG0N907617P ...

I had read online somewhere that ashift=13 works better for SSDs? Is not so? Could this be the reason why this pool is so slow? I didn't use an alternate ashift value when I created the other, faster pool of spinning rust.

I've read the sanoid docs and wiki but I couldn't see any guidelines or warnings about maximum number of datasets vs performance - that would be very handy . I'm sure I've heard Allan say ZFS should be able to handle around 10K snapshots per dataset before performance starts to degrade? It sounds like bcachefs is going to handle snapshots better than ZFS as Mr Overstreet says he has created 1M snapshots in his tests with no real performance degradation. I'll believe it when I see it.

Thanks