Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parallel deletion (without following symlinks) using jwalk #43

Open
Byron opened this issue Mar 15, 2020 · 0 comments
Open

parallel deletion (without following symlinks) using jwalk #43

Byron opened this issue Mar 15, 2020 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@Byron
Copy link
Owner

Byron commented Mar 15, 2020

Previously jwalk would follow symlinks, so we could not use it and instead implemented our own traversal.

Now with jwalk 0.5, this is supported and we can use our standard jwalker to do the job.

let mut files_or_dirs = vec![path];
let mut dirs = Vec::new();
let mut num_errors = 0;
while let Some(path) = files_or_dirs.pop() {
let assume_symlink_to_try_deletion = true;
let is_symlink = path
.symlink_metadata()
.map(|m| m.file_type().is_symlink())
.unwrap_or(assume_symlink_to_try_deletion);
if is_symlink {
// do not follow symlinks
num_errors += into_error_count(fs::remove_file(&path));
continue;
}
match fs::read_dir(&path) {
Ok(iterator) => {
dirs.push(path);
for entry in iterator {
match entry.map_err(io_err_to_usize) {
Ok(entry) => files_or_dirs.push(entry.path()),
Err(c) => num_errors += c,
}
}
}
Err(ref e) if e.kind() == io::ErrorKind::Other => {
// assume file, save IOps
num_errors += into_error_count(fs::remove_file(path));
continue;
}
Err(_) => {
num_errors += 1;
continue;
}
};
}

The function above would have to change to use jwalk. It should be straightforward to obtain a walker with this function:

dua-cli/src/common.rs

Lines 162 to 184 in 0d6116e

impl WalkOptions {
pub(crate) fn iter_from_path(&self, path: &Path) -> WalkDir {
WalkDir::new(path)
.follow_links(false)
.sort(match self.sorting {
TraversalSorting::None => false,
TraversalSorting::AlphabeticalByFileName => true,
})
.skip_hidden(false)
.process_read_dir(|_, dir_entry_results| {
dir_entry_results.iter_mut().for_each(|dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
dir_entry.client_state = Some(dir_entry.metadata());
}
})
})
.parallelism(if self.threads == 0 {
jwalk::Parallelism::RayonDefaultPool
} else {
jwalk::Parallelism::RayonNewPool(self.threads)
})
}
}

@Byron Byron added help wanted Extra attention is needed good first issue Good for newcomers labels Mar 15, 2020
@Byron Byron added the enhancement New feature or request label Jul 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant