Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

segment ring for fair pruning #7343

Open
Tracked by #8303
emhane opened this issue Mar 26, 2024 · 1 comment · May be fixed by #8180
Open
Tracked by #8303

segment ring for fair pruning #7343

emhane opened this issue Mar 26, 2024 · 1 comment · May be fixed by #8180
Labels
A-pruning Related to pruning or full node C-enhancement New feature or request M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity

Comments

@emhane
Copy link
Member

emhane commented Mar 26, 2024

Describe the feature

Segment pruning always starts with the same segment. Last segments in the iterator, risk not being pruned if the prune limit is small so it's often reached before pruning is done. We should implement a ring here: add a field to Pruner where we save the index in the chained segments iterator, of the last pruned segment. Upon calling prune_segments next time, the iterator should skip until the saved index + 1.

reth/crates/prune/src/pruner.rs

Lines 141 to 145 in c04dbe6

let static_file_segments = self.static_file_segments();
let segments = static_file_segments
.iter()
.map(|segment| (segment, PrunePurpose::StaticFile))
.chain(self.segments.iter().map(|segment| (segment, PrunePurpose::User)));

This will probably require modifying static_file_segments method, since it's important that no segments that need pruning are dropped when the iterator skips ahead to the last pruned segment.

reth/crates/prune/src/pruner.rs

Lines 215 to 243 in c04dbe6

/// Returns pre-configured segments that needs to be pruned according to the highest
/// static_files for [PruneSegment::Transactions], [PruneSegment::Headers] and
/// [PruneSegment::Receipts].
fn static_file_segments(&self) -> Vec<Box<dyn Segment<DB>>> {
let mut segments = Vec::<Box<dyn Segment<DB>>>::new();
let static_file_provider = self.provider_factory.static_file_provider();
if let Some(to_block) =
static_file_provider.get_highest_static_file_block(StaticFileSegment::Transactions)
{
segments
.push(Box::new(segments::Transactions::new(PruneMode::before_inclusive(to_block))))
}
if let Some(to_block) =
static_file_provider.get_highest_static_file_block(StaticFileSegment::Headers)
{
segments.push(Box::new(segments::Headers::new(PruneMode::before_inclusive(to_block))))
}
if let Some(to_block) =
static_file_provider.get_highest_static_file_block(StaticFileSegment::Receipts)
{
segments.push(Box::new(segments::Receipts::new(PruneMode::before_inclusive(to_block))))
}
segments
}

Additional context

No response
@emhane emhane added C-bug An unexpected or incorrect behavior A-pruning Related to pruning or full node labels Mar 26, 2024
@emhane emhane mentioned this issue Mar 26, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open for 21 days with no activity.

@github-actions github-actions bot added the S-stale This issue/PR is stale and will close with no further activity label Apr 17, 2024
@onbjerg onbjerg removed the S-stale This issue/PR is stale and will close with no further activity label Apr 17, 2024
@emhane emhane added the M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity label Apr 17, 2024
@emhane emhane changed the title chore(prune): segment ring for fair pruning segment ring for fair pruning May 8, 2024
@emhane emhane linked a pull request May 9, 2024 that will close this issue
Open
@shekhirin shekhirin added C-enhancement New feature or request and removed C-bug An unexpected or incorrect behavior labels May 9, 2024
@emhane emhane mentioned this issue May 18, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-pruning Related to pruning or full node C-enhancement New feature or request M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity
Projects
Reth Tracker
Status: Todo
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(pruning): fair pruning paradigmxyz/reth
3 participants
@shekhirin @onbjerg @emhane