New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit size of archives put to HSI #2

Open

3 of 4 tasks

kc9jud opened this issue Oct 4, 2020 · 1 comment

Labels

Contributor

kc9jud commented Oct 4, 2020 •

edited

According to the NERSC HPSS documentation (https://docs.nersc.gov/filesystems/archive/#avoid-very-large-files), files over 2TB are inefficient when put to HPSS. They recommend breaking files up into 500GB chunks if they get over that limit.

The hsi handler mcscript.task.archive_handler_hsi() should:

Inspect the size of an archive file to put.
Put the file directly using hsi if smaller than threshold, or
use split to break up the file into smaller segments and put those with hsi.

In addition (so that consumers of archives don't need to be aware of this splitting behavior):

mcscript should also provide a wrapper function to fetch and reassemble archives,

The text was updated successfully, but these errors were encountered:

kc9jud added the enhancement label

Contributor Author

kc9jud commented Oct 9, 2020

c849af4 implements the logic in archive_handler_hsi()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment