Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for analysis processes to be interrupted #37

Open
1313e opened this issue Aug 6, 2020 · 0 comments
Open

Allow for analysis processes to be interrupted #37

1313e opened this issue Aug 6, 2020 · 0 comments
Assignees
Labels
high priority This issue has a high priority maintenance Optimization/improvement/extension

Comments

@1313e
Copy link
Owner

1313e commented Aug 6, 2020

For very large emulators, it can sometimes still take many hours (or days, if they get extremely large) to analyze an iteration.
Losing progress in this analysis process due to an interruption (like, timeout on a cluster) could cost a lot of time.
This should be avoided.

I could modify the _evaluate_sam_set method to automatically make a backup of all evaluation progress whenever the entire plausible sample set has been evaluated in a single iteration.
I do not think it can be done any earlier due to that requiring many MPI calls back-and-forth, which heavily slows down the process.
If this modification could be made to specifically work for analyze, the method can check if an analysis process was in progress and continue if so.
I will have to think about what a KeyboardInterrupt does (do I see that as the user not wanting the process to finish and thus PRISM should remove the progress backup?).

@1313e 1313e added high priority This issue has a high priority maintenance Optimization/improvement/extension labels Aug 6, 2020
@1313e 1313e self-assigned this Aug 6, 2020
@1313e 1313e added the todo This will be worked on in the future label Aug 6, 2020
@1313e 1313e removed the todo This will be worked on in the future label Oct 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority This issue has a high priority maintenance Optimization/improvement/extension
Projects
None yet
Development

No branches or pull requests

1 participant