You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Current [groom] processing patterns create i/o bottlenecks. The various steps are carried out separately. Each step reads one file at a time, runs a command on that file, and writes the result. However, for most of these processes a file could be read and all steps executed while it's in memory, and the result written at the end.
Furthermore, the process could be carried out in parallel (especially if significant processing is required for any step and it the step has no function-level parallelism).
One reason to execute the steps independently is that there is sometimes usefulness in looking at mid-process results. We should consider a way to accomplish both application-level parallelism as well as how one might consider these mid-process results.
This is not urgent, but should be on our radar.
The text was updated successfully, but these errors were encountered:
Current [groom] processing patterns create i/o bottlenecks. The various steps are carried out separately. Each step reads one file at a time, runs a command on that file, and writes the result. However, for most of these processes a file could be read and all steps executed while it's in memory, and the result written at the end.
Furthermore, the process could be carried out in parallel (especially if significant processing is required for any step and it the step has no function-level parallelism).
One reason to execute the steps independently is that there is sometimes usefulness in looking at mid-process results. We should consider a way to accomplish both application-level parallelism as well as how one might consider these mid-process results.
This is not urgent, but should be on our radar.
The text was updated successfully, but these errors were encountered: