walredo metrics: some are just for apply_batch_postgres, others include apply_batch_neon

[problame]

Problem

apply_batch_postgres does .observe() on these metrics

neon/pageserver/src/walredo.rs

Lines 258 to 260 in 2d5a846

	WAL_REDO_TIME.observe(duration.as_secs_f64());
	WAL_REDO_RECORDS_HISTOGRAM.observe(len as f64);
	WAL_REDO_BYTES_HISTOGRAM.observe(nbytes as f64);

whereas apply_batch_neon does .observe() only on this metric

neon/pageserver/src/walredo.rs

Lines 350 to 352 in 2d5a846

	// FIXME: using the same metric here creates a bimodal distribution by default, and because
	// there could be multiple batch sizes this would be N+1 modal.
	WAL_REDO_TIME.observe(duration.as_secs_f64());

This makes it error-prone to reason about walredo metrics, as, e.g., WAL_REDO_TIME has a different _count than the WAL_REDO_RECORDS_HISTOGRAM and WAL_REDO_BYTES_HISTOGRAM

I stumbled across this and wasted multiple hours when trying to qualify async walredo.
(Actually, Joonas added that FIXME in the code quoted above. This issue addresses the FIXME)

Solution

1. For metrics that make sense for both functions (WAL_REDO_TIME, WAL_REDO_RECORDS_HISTOGRAM)
2. Group them together under a struct metrics::WalRedoManagerMetrics
3. observe them in both functions and add a label called replayer=walredoproc|inprocess to distinguish them.
4. The WAL_REDO_BYTES_HISTOGRAM is walredo process specific
5. Group it under a struct metrics::WalRedoProcessMetrics (maybe we already have something like that)
6. Rename the metric from pageserver_wal_redo_bytes_histogram to pageserver_wal_redo_proc_bytes_histogram

[koivunej]

Regarding my FIXME I think we concluded yesterday that the batch size does not matter for these metrics, because the time taken will be a big component of actual request completion.