Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve current [groom] processing patterns #480

Closed
cchriste opened this issue May 7, 2020 · 3 comments
Closed

improve current [groom] processing patterns #480

cchriste opened this issue May 7, 2020 · 3 comments
Assignees

Comments

@cchriste
Copy link
Contributor

cchriste commented May 7, 2020

Current [groom] processing patterns create i/o bottlenecks. The various steps are carried out separately. Each step reads one file at a time, runs a command on that file, and writes the result. However, for most of these processes a file could be read and all steps executed while it's in memory, and the result written at the end.

Furthermore, the process could be carried out in parallel (especially if significant processing is required for any step and it the step has no function-level parallelism).

One reason to execute the steps independently is that there is sometimes usefulness in looking at mid-process results. We should consider a way to accomplish both application-level parallelism as well as how one might consider these mid-process results.

This is not urgent, but should be on our radar.

@sheryjoe
Copy link
Contributor

Related to #865

@sheryjoe sheryjoe modified the milestones: 6.0 Leftover, 6.1 Release (May 2021) Feb 15, 2021
@cchriste
Copy link
Contributor Author

This is essentially how/why we want to get rid of things like GroomUtils, for which we may already have independent issues.

@jadie1
Copy link
Contributor

jadie1 commented Jun 9, 2021

Done with PR #1181

@jadie1 jadie1 closed this as completed Jun 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants