Something is wrong with the ledger shrinking

[diman-io]

Problem

Creating snapshot of slot 246464040 in snapshots/
Unable to process blockstore from starting slot 246460069 to 246464040; the blockstore does not contain a replayable chain between these slots. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found.
---
incremental-snapshot-246444769-246460069-Er4cZCqKufPKVzPWf3WUtV3N8MABj4gsnEMVnxRLBt4y.tar.zst
---
Ledger has data for 2877 slots 246461183 to 246478848
  with 2776 rooted slots from 246461183 to 246464006
  and 70 slots past the last root
---
            "--limit-ledger-size",
            "50000000",
            "--incremental-snapshot-interval-slots",
            "5000",
            "--rocksdb-shred-compaction",
            "fifo",

I received this on all three machines (one voting, two backup).
The validator continued to operate for ~2 hours after the cluster stopped until the attempt to create a snapshot.

[steviez]

As mentioned on Discord, I am highly suspect of this:

Ledger has data for 2877 slots 246461183 to 246478848

Running a node with --limit-ledger-size 50000000 will make the ledger retain ~50k slots (~1k shreds / slot) or about 5.5 hours of wallclock time. Your node's range spans ~17k slots (~2 hours of wallclock time).

The only scenario in which I'd expect your node to have so few slots is if you had just spun it up from scratch (no pre-existing ledger). Your node does not appear to be reporting metrics, so I can't do much digging unfortunately. If you still have your logs, grepping for below might give us something to go off of:

cleanup_service

[diman-io]

@steviez

Your node does not appear to be reporting metrics

You could check 7Ad8oaoaVPu9A1fZsUqMAiEoHGZCfGoJ5St1j1oBLNtq.
It's v1.16 (it was second backup from 1.16->1.17 update). But it has the same config.

[2024-02-06T15:55:59.706467145Z INFO  solana_ledger_tool] solana-ledger-tool 1.16.23 (src:4ac46ef5; feat:4033350765, client:SolanaLabs)

Ledger has data for 2877 slots 246461183 to 246478848
  with 2776 rooted slots from 246461183 to 246464006
  and 70 slots past the last root

144G	ledger/rocksdb_fifo/

Anyway, bellow 2 logs (grep cleanup_service | tail -n 100)

1. from voting node

[2024-02-06T08:24:33.293344083Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246451410, last_purge_slot=246450897, purge_interval=512, disk_utilization=Ok(152955439912)
[2024-02-06T08:24:33.293780045Z INFO  solana_core::ledger_cleanup_service] 60238390 alive shreds in slots [246448422, 246451443], mean of 19933 shreds per slot
[2024-02-06T08:24:33.293995156Z INFO  solana_core::ledger_cleanup_service] purging data older than 246448935
[2024-02-06T08:24:33.294169837Z INFO  solana_core::ledger_cleanup_service] purge_slots took 159us
[2024-02-06T08:28:12.548188217Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246451923, last_purge_slot=246451410, purge_interval=512, disk_utilization=Ok(152498938243)
[2024-02-06T08:28:12.548748140Z INFO  solana_core::ledger_cleanup_service] 60238380 alive shreds in slots [246448936, 246451954], mean of 19953 shreds per slot
[2024-02-06T08:28:12.548896811Z INFO  solana_core::ledger_cleanup_service] purging data older than 246449449
[2024-02-06T08:28:12.550356399Z INFO  solana_core::ledger_cleanup_service] purge_slots took 1ms
[2024-02-06T08:31:46.541374476Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246452436, last_purge_slot=246451923, purge_interval=512, disk_utilization=Ok(152361686609)
[2024-02-06T08:31:46.541828238Z INFO  solana_core::ledger_cleanup_service] 60238378 alive shreds in slots [246449450, 246452467], mean of 19959 shreds per slot
[2024-02-06T08:31:46.541965759Z INFO  solana_core::ledger_cleanup_service] purging data older than 246449962
[2024-02-06T08:31:46.542135850Z INFO  solana_core::ledger_cleanup_service] purge_slots took 157us
[2024-02-06T08:35:23.161268364Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246452949, last_purge_slot=246452436, purge_interval=512, disk_utilization=Ok(152628766406)
[2024-02-06T08:35:23.161668856Z INFO  solana_core::ledger_cleanup_service] 60238378 alive shreds in slots [246449963, 246452982], mean of 19946 shreds per slot
[2024-02-06T08:35:23.161850317Z INFO  solana_core::ledger_cleanup_service] purging data older than 246450476
[2024-02-06T08:35:23.163072763Z INFO  solana_core::ledger_cleanup_service] purge_slots took 1ms
[2024-02-06T08:39:04.843922800Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246453462, last_purge_slot=246452949, purge_interval=512, disk_utilization=Ok(152308133208)
[2024-02-06T08:39:04.844444133Z INFO  solana_core::ledger_cleanup_service] 60238383 alive shreds in slots [246450477, 246453499], mean of 19926 shreds per slot
[2024-02-06T08:39:04.844580914Z INFO  solana_core::ledger_cleanup_service] purging data older than 246450990
[2024-02-06T08:39:04.846064162Z INFO  solana_core::ledger_cleanup_service] purge_slots took 1ms
[2024-02-06T08:42:47.372462102Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246453975, last_purge_slot=246453462, purge_interval=512, disk_utilization=Ok(152543360118)
[2024-02-06T08:42:47.372945344Z INFO  solana_core::ledger_cleanup_service] 60238365 alive shreds in slots [246450991, 246454007], mean of 19966 shreds per slot
[2024-02-06T08:42:47.373059875Z INFO  solana_core::ledger_cleanup_service] purging data older than 246451503
[2024-02-06T08:42:47.375093956Z INFO  solana_core::ledger_cleanup_service] purge_slots took 2ms
[2024-02-06T08:46:28.714888601Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246454488, last_purge_slot=246453975, purge_interval=512, disk_utilization=Ok(152749935669)
[2024-02-06T08:46:28.715336473Z INFO  solana_core::ledger_cleanup_service] 60238363 alive shreds in slots [246451504, 246454519], mean of 19972 shreds per slot
[2024-02-06T08:46:28.715463974Z INFO  solana_core::ledger_cleanup_service] purging data older than 246452016
[2024-02-06T08:46:28.716933322Z INFO  solana_core::ledger_cleanup_service] purge_slots took 1ms
[2024-02-06T08:50:06.996633054Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246455001, last_purge_slot=246454488, purge_interval=512, disk_utilization=Ok(152476824608)
[2024-02-06T08:50:06.997055557Z INFO  solana_core::ledger_cleanup_service] 60270311 alive shreds in slots [246452017, 246455034], mean of 19970 shreds per slot
[2024-02-06T08:50:06.997175927Z INFO  solana_core::ledger_cleanup_service] purging data older than 246452531
[2024-02-06T08:50:06.997337388Z INFO  solana_core::ledger_cleanup_service] purge_slots took 151us
[2024-02-06T08:53:38.508286640Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246455514, last_purge_slot=246455001, purge_interval=512, disk_utilization=Ok(152452007527)
[2024-02-06T08:53:38.508751713Z INFO  solana_core::ledger_cleanup_service] 60270314 alive shreds in slots [246452532, 246455546], mean of 19990 shreds per slot
[2024-02-06T08:53:38.508869504Z INFO  solana_core::ledger_cleanup_service] purging data older than 246453045
[2024-02-06T08:53:38.510036870Z INFO  solana_core::ledger_cleanup_service] purge_slots took 1ms
[2024-02-06T08:57:13.970150514Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246456027, last_purge_slot=246455514, purge_interval=512, disk_utilization=Ok(152608278451)
[2024-02-06T08:57:13.970655816Z INFO  solana_core::ledger_cleanup_service] 60270314 alive shreds in slots [246453046, 246456062], mean of 19976 shreds per slot
[2024-02-06T08:57:13.970784776Z INFO  solana_core::ledger_cleanup_service] purging data older than 246453560
[2024-02-06T08:57:13.970992926Z INFO  solana_core::ledger_cleanup_service] purge_slots took 187us
[2024-02-06T09:00:44.459276366Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246456540, last_purge_slot=246456027, purge_interval=512, disk_utilization=Ok(152897202839)
[2024-02-06T09:00:44.459881868Z INFO  solana_core::ledger_cleanup_service] 60270313 alive shreds in slots [246453561, 246456572], mean of 20010 shreds per slot
[2024-02-06T09:00:44.460007288Z INFO  solana_core::ledger_cleanup_service] purging data older than 246454074
[2024-02-06T09:00:44.472195601Z INFO  solana_core::ledger_cleanup_service] purge_slots took 12ms
[2024-02-06T09:04:15.568459294Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246457053, last_purge_slot=246456540, purge_interval=512, disk_utilization=Ok(151701867978)
[2024-02-06T09:04:15.568950116Z INFO  solana_core::ledger_cleanup_service] 60270312 alive shreds in slots [246454075, 246457086], mean of 20010 shreds per slot
[2024-02-06T09:04:15.569056537Z INFO  solana_core::ledger_cleanup_service] purging data older than 246454588
[2024-02-06T09:04:15.579766969Z INFO  solana_core::ledger_cleanup_service] purge_slots took 10ms
[2024-02-06T09:07:48.185981018Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246457566, last_purge_slot=246457053, purge_interval=512, disk_utilization=Ok(152170461635)
[2024-02-06T09:07:48.186406730Z INFO  solana_core::ledger_cleanup_service] 60270310 alive shreds in slots [246454589, 246457597], mean of 20030 shreds per slot
[2024-02-06T09:07:48.186511940Z INFO  solana_core::ledger_cleanup_service] purging data older than 246455101
[2024-02-06T09:07:48.186655411Z INFO  solana_core::ledger_cleanup_service] purge_slots took 134us
[2024-02-06T09:11:26.968557953Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246458079, last_purge_slot=246457566, purge_interval=512, disk_utilization=Ok(153197709342)
[2024-02-06T09:11:26.969002055Z INFO  solana_core::ledger_cleanup_service] 60270310 alive shreds in slots [246455102, 246458115], mean of 19996 shreds per slot
[2024-02-06T09:11:26.969128376Z INFO  solana_core::ledger_cleanup_service] purging data older than 246455615
[2024-02-06T09:11:26.969317617Z INFO  solana_core::ledger_cleanup_service] purge_slots took 171us
[2024-02-06T09:15:06.258191723Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246458592, last_purge_slot=246458079, purge_interval=512, disk_utilization=Ok(153240914660)
[2024-02-06T09:15:06.258766636Z INFO  solana_core::ledger_cleanup_service] 60270310 alive shreds in slots [246455616, 246458623], mean of 20036 shreds per slot
[2024-02-06T09:15:06.258892117Z INFO  solana_core::ledger_cleanup_service] purging data older than 246456128
[2024-02-06T09:15:06.263789009Z INFO  solana_core::ledger_cleanup_service] purge_slots took 4ms
[2024-02-06T09:18:45.862391759Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246459105, last_purge_slot=246458592, purge_interval=512, disk_utilization=Ok(151581131261)
[2024-02-06T09:18:45.862875381Z INFO  solana_core::ledger_cleanup_service] 60270313 alive shreds in slots [246456129, 246459138], mean of 20023 shreds per slot
[2024-02-06T09:18:45.862997652Z INFO  solana_core::ledger_cleanup_service] purging data older than 246456641
[2024-02-06T09:18:45.863141282Z INFO  solana_core::ledger_cleanup_service] purge_slots took 135us
[2024-02-06T09:22:16.724197570Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246459618, last_purge_slot=246459105, purge_interval=512, disk_utilization=Ok(152437789757)
[2024-02-06T09:22:16.724579871Z INFO  solana_core::ledger_cleanup_service] 60270310 alive shreds in slots [246456642, 246459649], mean of 20036 shreds per slot
[2024-02-06T09:22:16.724685912Z INFO  solana_core::ledger_cleanup_service] purging data older than 246457154
[2024-02-06T09:22:16.740637707Z INFO  solana_core::ledger_cleanup_service] purge_slots took 15ms
[2024-02-06T09:25:51.726354467Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246460131, last_purge_slot=246459618, purge_interval=512, disk_utilization=Ok(152620295072)
[2024-02-06T09:25:51.726925330Z INFO  solana_core::ledger_cleanup_service] 60270329 alive shreds in slots [246457155, 246460162], mean of 20036 shreds per slot
[2024-02-06T09:25:51.727043121Z INFO  solana_core::ledger_cleanup_service] purging data older than 246457667
[2024-02-06T09:25:51.739257478Z INFO  solana_core::ledger_cleanup_service] purge_slots took 12ms
[2024-02-06T09:29:22.634069369Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246460644, last_purge_slot=246460131, purge_interval=512, disk_utilization=Ok(153425057554)
[2024-02-06T09:29:22.634590342Z INFO  solana_core::ledger_cleanup_service] 60270342 alive shreds in slots [246457668, 246460675], mean of 20036 shreds per slot
[2024-02-06T09:29:22.634725210Z INFO  solana_core::ledger_cleanup_service] purging data older than 246458180
[2024-02-06T09:29:22.634914468Z INFO  solana_core::ledger_cleanup_service] purge_slots took 174us
[2024-02-06T09:32:56.702455837Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246461157, last_purge_slot=246460644, purge_interval=512, disk_utilization=Ok(151944320378)
[2024-02-06T09:32:56.702974803Z INFO  solana_core::ledger_cleanup_service] 60270342 alive shreds in slots [246458181, 246461189], mean of 20030 shreds per slot
[2024-02-06T09:32:56.703099272Z INFO  solana_core::ledger_cleanup_service] purging data older than 246458693
[2024-02-06T09:32:56.703236630Z INFO  solana_core::ledger_cleanup_service] purge_slots took 129us
[2024-02-06T09:36:33.479616457Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246461670, last_purge_slot=246461157, purge_interval=512, disk_utilization=Ok(152049254449)
[2024-02-06T09:36:33.480215384Z INFO  solana_core::ledger_cleanup_service] 60270358 alive shreds in slots [246458694, 246461706], mean of 20003 shreds per slot
[2024-02-06T09:36:33.480319554Z INFO  solana_core::ledger_cleanup_service] purging data older than 246459207
[2024-02-06T09:36:33.486181966Z INFO  solana_core::ledger_cleanup_service] purge_slots took 5ms
[2024-02-06T09:40:12.776994113Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246462183, last_purge_slot=246461670, purge_interval=512, disk_utilization=Ok(153206169893)
[2024-02-06T09:40:12.777422842Z INFO  solana_core::ledger_cleanup_service] 60270347 alive shreds in slots [246459208, 246462219], mean of 20010 shreds per slot
[2024-02-06T09:40:12.777585072Z INFO  solana_core::ledger_cleanup_service] purging data older than 246459721
[2024-02-06T09:40:12.777762601Z INFO  solana_core::ledger_cleanup_service] purge_slots took 166us
[2024-02-06T09:43:48.384659001Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246462696, last_purge_slot=246462183, purge_interval=512, disk_utilization=Ok(152498602727)
[2024-02-06T09:43:48.385147161Z INFO  solana_core::ledger_cleanup_service] 60270348 alive shreds in slots [246459722, 246462727], mean of 20050 shreds per slot
[2024-02-06T09:43:48.385271761Z INFO  solana_core::ledger_cleanup_service] purging data older than 246460234
[2024-02-06T09:43:48.393043976Z INFO  solana_core::ledger_cleanup_service] purge_slots took 7ms
[2024-02-06T09:47:24.652401560Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246463212, last_purge_slot=246462696, purge_interval=512, disk_utilization=Ok(152199820885)
[2024-02-06T09:47:24.652863530Z INFO  solana_core::ledger_cleanup_service] 60270347 alive shreds in slots [246460235, 246463246], mean of 20010 shreds per slot
[2024-02-06T09:47:24.652958200Z INFO  solana_core::ledger_cleanup_service] purging data older than 246460748
[2024-02-06T09:47:24.667259265Z INFO  solana_core::ledger_cleanup_service] purge_slots took 14ms
[2024-02-06T09:51:01.102944519Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246463725, last_purge_slot=246463212, purge_interval=512, disk_utilization=Ok(153328525431)
[2024-02-06T09:51:01.103481320Z INFO  solana_core::ledger_cleanup_service] 60270351 alive shreds in slots [246460749, 246463758], mean of 20023 shreds per slot
[2024-02-06T09:51:01.103600400Z INFO  solana_core::ledger_cleanup_service] purging data older than 246461261
[2024-02-06T09:51:01.109773606Z INFO  solana_core::ledger_cleanup_service] purge_slots took 6ms

2. from backup node 7Ad8oaoaVPu9A1fZsUqMAiEoHGZCfGoJ5St1j1oBLNtq

[2024-02-06T08:23:33.971777647Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246451272, last_purge_slot=246450759, purge_interval=512, disk_utilization=Ok(154307161210)
[2024-02-06T08:23:33.972266207Z INFO  solana_core::ledger_cleanup_service] 60518872 alive shreds in slots [246448357, 246451303], mean of 20535 shreds per slot
[2024-02-06T08:23:33.972385502Z INFO  solana_core::ledger_cleanup_service] purging data older than 246448869
[2024-02-06T08:23:33.972547797Z INFO  solana_core::ledger_cleanup_service] purge_slots took 148us
[2024-02-06T08:27:12.411049339Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246451785, last_purge_slot=246451272, purge_interval=512, disk_utilization=Ok(154071597398)
[2024-02-06T08:27:12.411547086Z INFO  solana_core::ledger_cleanup_service] 60518868 alive shreds in slots [246448870, 246451817], mean of 20528 shreds per slot
[2024-02-06T08:27:12.411667383Z INFO  solana_core::ledger_cleanup_service] purging data older than 246449382
[2024-02-06T08:27:12.425134064Z INFO  solana_core::ledger_cleanup_service] purge_slots took 13ms
[2024-02-06T08:30:49.989907209Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246452298, last_purge_slot=246451785, purge_interval=512, disk_utilization=Ok(153924436510)
[2024-02-06T08:30:49.990588852Z INFO  solana_core::ledger_cleanup_service] 60518857 alive shreds in slots [246449383, 246452330], mean of 20528 shreds per slot
[2024-02-06T08:30:49.990719608Z INFO  solana_core::ledger_cleanup_service] purging data older than 246449895
[2024-02-06T08:30:49.990862707Z INFO  solana_core::ledger_cleanup_service] purge_slots took 132us
[2024-02-06T08:34:23.496978407Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246452811, last_purge_slot=246452298, purge_interval=512, disk_utilization=Ok(153330767976)
[2024-02-06T08:34:23.497652054Z INFO  solana_core::ledger_cleanup_service] 60518859 alive shreds in slots [246449896, 246452843], mean of 20528 shreds per slot
[2024-02-06T08:34:23.497802357Z INFO  solana_core::ledger_cleanup_service] purging data older than 246450408
[2024-02-06T08:34:23.517701553Z INFO  solana_core::ledger_cleanup_service] purge_slots took 19ms
[2024-02-06T08:38:04.437856263Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246453324, last_purge_slot=246452811, purge_interval=512, disk_utilization=Ok(153096926904)
[2024-02-06T08:38:04.438281203Z INFO  solana_core::ledger_cleanup_service] 60518866 alive shreds in slots [246450409, 246453355], mean of 20535 shreds per slot
[2024-02-06T08:38:04.438420625Z INFO  solana_core::ledger_cleanup_service] purging data older than 246450921
[2024-02-06T08:38:04.438622875Z INFO  solana_core::ledger_cleanup_service] purge_slots took 183us
[2024-02-06T08:41:46.620288087Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246453837, last_purge_slot=246453324, purge_interval=512, disk_utilization=Ok(154266313254)
[2024-02-06T08:41:46.620729898Z INFO  solana_core::ledger_cleanup_service] 60518866 alive shreds in slots [246450922, 246453869], mean of 20528 shreds per slot
[2024-02-06T08:41:46.620862408Z INFO  solana_core::ledger_cleanup_service] purging data older than 246451434
[2024-02-06T08:41:46.621037908Z INFO  solana_core::ledger_cleanup_service] purge_slots took 162us
[2024-02-06T08:45:32.173504459Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246454350, last_purge_slot=246453837, purge_interval=512, disk_utilization=Ok(154336194997)
[2024-02-06T08:45:32.174067919Z INFO  solana_core::ledger_cleanup_service] 60518866 alive shreds in slots [246451435, 246454385], mean of 20507 shreds per slot
[2024-02-06T08:45:32.174183487Z INFO  solana_core::ledger_cleanup_service] purging data older than 246451947
[2024-02-06T08:45:32.188496132Z INFO  solana_core::ledger_cleanup_service] purge_slots took 14ms
[2024-02-06T08:49:07.061654455Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246454863, last_purge_slot=246454350, purge_interval=512, disk_utilization=Ok(154682750150)
[2024-02-06T08:49:07.062161188Z INFO  solana_core::ledger_cleanup_service] 60518852 alive shreds in slots [246451948, 246454895], mean of 20528 shreds per slot
[2024-02-06T08:49:07.062289961Z INFO  solana_core::ledger_cleanup_service] purging data older than 246452460
[2024-02-06T08:49:07.062475800Z INFO  solana_core::ledger_cleanup_service] purge_slots took 177us
[2024-02-06T08:52:41.859716451Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246455376, last_purge_slot=246454863, purge_interval=512, disk_utilization=Ok(154163024335)
[2024-02-06T08:52:41.860219558Z INFO  solana_core::ledger_cleanup_service] 60518856 alive shreds in slots [246452461, 246455407], mean of 20535 shreds per slot
[2024-02-06T08:52:41.860331879Z INFO  solana_core::ledger_cleanup_service] purging data older than 246452973
[2024-02-06T08:52:41.860492602Z INFO  solana_core::ledger_cleanup_service] purge_slots took 147us
[2024-02-06T08:56:15.023377365Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246455889, last_purge_slot=246455376, purge_interval=512, disk_utilization=Ok(154038796166)
[2024-02-06T08:56:15.023952868Z INFO  solana_core::ledger_cleanup_service] 60518840 alive shreds in slots [246452974, 246455921], mean of 20528 shreds per slot
[2024-02-06T08:56:15.024091199Z INFO  solana_core::ledger_cleanup_service] purging data older than 246453486
[2024-02-06T08:56:15.024293890Z INFO  solana_core::ledger_cleanup_service] purge_slots took 187us
[2024-02-06T08:59:47.583449388Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246456402, last_purge_slot=246455889, purge_interval=512, disk_utilization=Ok(152226441813)
[2024-02-06T08:59:47.584027576Z INFO  solana_core::ledger_cleanup_service] 60518856 alive shreds in slots [246453487, 246456434], mean of 20528 shreds per slot
[2024-02-06T08:59:47.584208737Z INFO  solana_core::ledger_cleanup_service] purging data older than 246453999
[2024-02-06T08:59:47.596837928Z INFO  solana_core::ledger_cleanup_service] purge_slots took 12ms
[2024-02-06T09:03:18.401340535Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246456915, last_purge_slot=246456402, purge_interval=512, disk_utilization=Ok(153931362447)
[2024-02-06T09:03:18.401918323Z INFO  solana_core::ledger_cleanup_service] 60518856 alive shreds in slots [246454000, 246456947], mean of 20528 shreds per slot
[2024-02-06T09:03:18.402031576Z INFO  solana_core::ledger_cleanup_service] purging data older than 246454512
[2024-02-06T09:03:18.402188211Z INFO  solana_core::ledger_cleanup_service] purge_slots took 146us
[2024-02-06T09:06:49.926741424Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246457428, last_purge_slot=246456915, purge_interval=512, disk_utilization=Ok(151946873406)
[2024-02-06T09:06:49.927237667Z INFO  solana_core::ledger_cleanup_service] 60518842 alive shreds in slots [246454513, 246457459], mean of 20535 shreds per slot
[2024-02-06T09:06:49.927351351Z INFO  solana_core::ledger_cleanup_service] purging data older than 246455025
[2024-02-06T09:06:49.940329412Z INFO  solana_core::ledger_cleanup_service] purge_slots took 12ms
[2024-02-06T09:10:27.688490357Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246457941, last_purge_slot=246457428, purge_interval=512, disk_utilization=Ok(152729499703)
[2024-02-06T09:10:27.688998352Z INFO  solana_core::ledger_cleanup_service] 60518837 alive shreds in slots [246455026, 246457973], mean of 20528 shreds per slot
[2024-02-06T09:10:27.689103741Z INFO  solana_core::ledger_cleanup_service] purging data older than 246455538
[2024-02-06T09:10:27.689243453Z INFO  solana_core::ledger_cleanup_service] purge_slots took 133us
[2024-02-06T09:14:06.719744158Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246458454, last_purge_slot=246457941, purge_interval=512, disk_utilization=Ok(154248296007)
[2024-02-06T09:14:06.720342193Z INFO  solana_core::ledger_cleanup_service] 60518831 alive shreds in slots [246455539, 246458486], mean of 20528 shreds per slot
[2024-02-06T09:14:06.720476045Z INFO  solana_core::ledger_cleanup_service] purging data older than 246456051
[2024-02-06T09:14:06.720638600Z INFO  solana_core::ledger_cleanup_service] purge_slots took 148us
[2024-02-06T09:17:46.132872535Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246458967, last_purge_slot=246458454, purge_interval=512, disk_utilization=Ok(153169558097)
[2024-02-06T09:17:46.133416498Z INFO  solana_core::ledger_cleanup_service] 60518822 alive shreds in slots [246456052, 246458999], mean of 20528 shreds per slot
[2024-02-06T09:17:46.133538518Z INFO  solana_core::ledger_cleanup_service] purging data older than 246456564
[2024-02-06T09:17:46.147258658Z INFO  solana_core::ledger_cleanup_service] purge_slots took 13ms
[2024-02-06T09:21:19.035004823Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246459480, last_purge_slot=246458967, purge_interval=512, disk_utilization=Ok(153450202362)
[2024-02-06T09:21:19.035434802Z INFO  solana_core::ledger_cleanup_service] 60518822 alive shreds in slots [246456565, 246459512], mean of 20528 shreds per slot
[2024-02-06T09:21:19.035606395Z INFO  solana_core::ledger_cleanup_service] purging data older than 246457077
[2024-02-06T09:21:19.035763871Z INFO  solana_core::ledger_cleanup_service] purge_slots took 146us
[2024-02-06T09:24:54.258999750Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246459993, last_purge_slot=246459480, purge_interval=512, disk_utilization=Ok(151940395856)
[2024-02-06T09:24:54.259520260Z INFO  solana_core::ledger_cleanup_service] 60518808 alive shreds in slots [246457078, 246460025], mean of 20528 shreds per slot
[2024-02-06T09:24:54.259636599Z INFO  solana_core::ledger_cleanup_service] purging data older than 246457590
[2024-02-06T09:24:54.266048965Z INFO  solana_core::ledger_cleanup_service] purge_slots took 6ms
[2024-02-06T09:28:26.603554117Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246460506, last_purge_slot=246459993, purge_interval=512, disk_utilization=Ok(152683996596)
[2024-02-06T09:28:26.604104854Z INFO  solana_core::ledger_cleanup_service] 60518805 alive shreds in slots [246457591, 246460538], mean of 20528 shreds per slot
[2024-02-06T09:28:26.604245047Z INFO  solana_core::ledger_cleanup_service] purging data older than 246458103
[2024-02-06T09:28:26.604419265Z INFO  solana_core::ledger_cleanup_service] purge_slots took 163us
[2024-02-06T09:31:57.707193264Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246461019, last_purge_slot=246460506, purge_interval=512, disk_utilization=Ok(154434002257)
[2024-02-06T09:31:57.707890446Z INFO  solana_core::ledger_cleanup_service] 60518826 alive shreds in slots [246458104, 246461051], mean of 20528 shreds per slot
[2024-02-06T09:31:57.708072859Z INFO  solana_core::ledger_cleanup_service] purging data older than 246458616
[2024-02-06T09:31:57.719233251Z INFO  solana_core::ledger_cleanup_service] purge_slots took 11ms
[2024-02-06T09:35:33.895019446Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246461532, last_purge_slot=246461019, purge_interval=512, disk_utilization=Ok(152736728762)
[2024-02-06T09:35:33.895408869Z INFO  solana_core::ledger_cleanup_service] 60518826 alive shreds in slots [246458617, 246461563], mean of 20535 shreds per slot
[2024-02-06T09:35:33.895516311Z INFO  solana_core::ledger_cleanup_service] purging data older than 246459129
[2024-02-06T09:35:33.902154824Z INFO  solana_core::ledger_cleanup_service] purge_slots took 6ms
[2024-02-06T09:39:11.249666030Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246462045, last_purge_slot=246461532, purge_interval=512, disk_utilization=Ok(153851192939)
[2024-02-06T09:39:11.250259457Z INFO  solana_core::ledger_cleanup_service] 60518826 alive shreds in slots [246459130, 246462077], mean of 20528 shreds per slot
[2024-02-06T09:39:11.250392207Z INFO  solana_core::ledger_cleanup_service] purging data older than 246459642
[2024-02-06T09:39:11.250556897Z INFO  solana_core::ledger_cleanup_service] purge_slots took 151us
[2024-02-06T09:42:51.509339591Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246462558, last_purge_slot=246462045, purge_interval=512, disk_utilization=Ok(154339590242)
[2024-02-06T09:42:51.509868918Z INFO  solana_core::ledger_cleanup_service] 60518829 alive shreds in slots [246459643, 246462590], mean of 20528 shreds per slot
[2024-02-06T09:42:51.509986649Z INFO  solana_core::ledger_cleanup_service] purging data older than 246460155
[2024-02-06T09:42:51.522924549Z INFO  solana_core::ledger_cleanup_service] purge_slots took 12ms
[2024-02-06T09:46:23.784538907Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246463071, last_purge_slot=246462558, purge_interval=512, disk_utilization=Ok(153063516241)
[2024-02-06T09:46:23.784964127Z INFO  solana_core::ledger_cleanup_service] 60518844 alive shreds in slots [246460156, 246463107], mean of 20500 shreds per slot
[2024-02-06T09:46:23.785099141Z INFO  solana_core::ledger_cleanup_service] purging data older than 246460669
[2024-02-06T09:46:23.785235528Z INFO  solana_core::ledger_cleanup_service] purge_slots took 128us
[2024-02-06T09:50:02.678614417Z INFO  solana_core::ledger_cleanup_service] purge: last_root=246463584, last_purge_slot=246463071, purge_interval=512, disk_utilization=Ok(153661945723)
[2024-02-06T09:50:02.679211972Z INFO  solana_core::ledger_cleanup_service] 60518836 alive shreds in slots [246460670, 246463615], mean of 20542 shreds per slot
[2024-02-06T09:50:02.679362706Z INFO  solana_core::ledger_cleanup_service] purging data older than 246461182
[2024-02-06T09:50:02.679535331Z INFO  solana_core::ledger_cleanup_service] purge_slots took 160us

[diman-io]

@steviez
Based on the logs and your comment, something is wrong either with the shredding count (~20K per slot) or with the expectations (1K per slot).

[diman-io]

I asked the guys from the Telegram group to do the same grep ~20K before the restart and now. I currently have ~500. But I deleted the ledger. I think they didn't delete it.
If someone else provides any significant response in Telegram, I'll add it.

[steviez]

@diman-io - Thanks for the responses, I have enough information / think I sufficiently understand the problem so probably no need to request / post any further reports

[diman-io]

@steviez hi, I was just wondering if there has been any progress with this issue?

[steviez]

@steviez hi, I was just wondering if there has been any progress with this issue?

Apologies, I just saw your comment in other issue and realized I never responded. After doing some testing on our side, I think we're actually going to deprecate FIFO as an option. The perf gains that it provided over the default are not as great after some optimization we did to the default (level)

I'm not 100% sure when we'll roll this out, but I would swap over when you get a chance