feat: run StaticFileProvider::check_consistency on start up

[joshieDo]

The bulk of changes are regarding ProviderFactoryinitialization, but the real change is introducing a function to check the overall consistency between static files and database:

reth/crates/storage/provider/src/providers/static_file/manager.rs

Lines 512 to 516 in 14e2dbc

	pub fn check_consistency<TX: DbTx>(
	&self,
	provider: &DatabaseProvider<TX>,
	has_receipt_pruning: bool,
	) -> ProviderResult<Option<PipelineTarget>> {

For each segment the following invariants are checked:

reth/crates/storage/provider/src/providers/static_file/manager.rs

Lines 611 to 621 in 14e2dbc

	/// Check invariants for each corresponding table and static file segment:
	///
	/// * the corresponding database table should overlap or have continuity in their keys
	/// ([TxNumber] or [BlockNumber]).
	/// * its highest block should match the stage checkpoint block number if it's equal or higher
	/// than the corresponding database table last entry.
	/// * If the checkpoint block is higher, then request a pipeline unwind to the static file
	/// block.
	/// * If the checkpoint block is lower, then heal by removing rows from the static file.
	fn ensure_invariants<TX: DbTx, T: Table<Key = u64>>(
	&self,

These will be called on starting up the node, before any data is queried from storage. If it fails any check it will try to self-heal, and if not possible, return a PipelineTarget::Unwind(N) that will be straight away handled by a standalone unwind-only pipeline.

The tests live in crates/stages/mod.rs because there are other necessary test functions there and can not be imported into providers crate. Follow-up will refactor the test-utils in stages crate and move them to providers crate.

Pending live testing - but open to review

[shekhirin]

can we move this to static file manager, so that we can re-use it in stages as well?

[shekhirin]

I made a first pass, want to also look at unwrap_or_default() usage

[shekhirin]

is it correct that if we don't even have a genesis block in the segment, we unwrap these to 0?

[mattsse]

a few nits

[mattsse]

looks like most of the changes concern the Providerfactory::new call

this no longer requires a result

reth/crates/storage/provider/src/providers/database/mod.rs

Lines 53 to 57 in 76156e2

	pub fn new(
	db: DB,
	chain_spec: Arc<ChainSpec>,
	static_file_provider: StaticFileProvider,
	) -> ProviderResult<ProviderFactory<DB>> {

[joshieDo]

looks like most of the changes concern the Providerfactory::new call

yes, because of the stacked PR: #8405