Pause IBD during AssumeUTXO snapshot load

[Sjors]

Please describe the feature you'd like to see added.

The loadtxoutset RPC call should cause IBD to pause until the snapshot is loaded and the snapshot chain is activated.

Is your feature related to a problem, if so please describe it.

When I start a fresh node and load a mainnet shapshot (#28553) any progress is completely buried in the log.

More importantly, I get the strong impression, though haven't properly tested this, that the IBD process slows down the snapshot load. This makes sense given the amount of I/O involved. It's definitely slower than the process of creating the snapshot.

On my AMD Ryzen 7950x machine the IBD reaches block 300,000 while the snapshot has only loaded 71.2% (where is triggers a cache flush). By the time 100% is loaded IBD is at 324,000. When the snapshot chainstate is saved IBD reaches 362,000.

Describe the solution you'd like

Pause IBD until the snapshot is loaded and the snapshot chain is activated.

Describe any alternatives you've considered

1. Do nothing
2. Keep IBD going but at low priority (how???)

Please leave any additional context

No response

[Sjors]

I loaded a mainnet snapshot today with a pruned node with limited --dbcache=10000 on a 2019 MacBook Pro. This time I had the node sync for about an hour before loading the snapshot.

At some point the log says:

2024-05-28T12:38:11Z Cache size (10375257872) exceeds total space (10370521497)
2024-05-28T12:38:11Z FlushSnapshotToDisk: flushing coins cache (10375 MB) started

This flush took 50 minutes, while the chain synced from 392,803 to 417,312. It then continues loading coins from the snapshot.

[jamesob]

In an earlier version of the code, we would acquire cs_main during the duration of snapshot load/activation. Maybe worth going back to that?

[Sjors]

IIUC taking cs_main blocks all RPC requests and the GUI, and this process can take half an hour or so on a modernish laptop. So I would prefer a less overkill method of pausing IBD. During my own testing I just turned the network off, but that's maybe not ideal either. Adding a few seconds delay before requesting new blocks might do the trick, while preventing our connections from timing out.