Workaround for independent writes to Iterations in parallel, better detection of BP5 which in turn uncovers more instances of the first issue #1619
+288
−67
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
franzpoeschel
force-pushed
the
fix-iteration-flush
branch
from
ce5bfc3
to
50fc0d1
Compare
franzpoeschel
force-pushed
the
fix-iteration-flush
branch
from
50fc0d1
to
17c42f3
Compare
franzpoeschel
force-pushed
the
fix-iteration-flush
branch
from
237b394
to
8bbc170
Compare
Somehow PerformDataWrite() leads to trouble with this pattern.
This reverts commit 36597bd. No longer needed after rebasing on fix-iteration-flush
It used Series::flush non-collectively
franzpoeschel
changed the title
franzpoeschel
commented
Jun 6, 2024
| @@ -946,10 +946,16 @@ void hipace_like_write(std::string const &file_ending) | |||
| int const last_step = 100; | |||
| int const my_first_step = i_mpi_rank * int(local_Nz); | |||
| int const all_last_step = last_step + (i_mpi_size - 1) * int(local_Nz); | |||
|
|
|||
| bool participate_in_barrier = true; | |||
There was a problem hiding this comment.
@ax3l Can you please check if this bug also affects Hipace? Currently, the sequence of barriers and flushes dont match from rank to rank. This was uncovered only now, since flushing is effectively not collective in many situations, but this test now uses adios2::Engine::PerformDataWrite() of BP5 which is a bit stricter there.
| "adios2.engine.preferred_flush_target = \"buffer\""); | ||
| int size, rank; | ||
| MPI_Comm_rank(MPI_COMM_WORLD, &rank); | ||
| MPI_Comm_size(MPI_COMM_WORLD, &size); |
Check notice
Code scanning / CodeQL
Commented-out code Note
franzpoeschel
force-pushed
the
fix-iteration-flush
branch
from
7b9d624
to
9dd2a14
Compare
franzpoeschel
force-pushed
the
fix-iteration-flush
branch
from
9dd2a14
to
72a465c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This somewhat fixes #1616 until we add a better solution. With this PR:
seriesFlush()will always flush the containing Iteration if called from within an Iteration (and will ignore missingdirtyannotations).At the same time, I added a better detection for BP5-specific features. Since this means that
adios2::Engine::PerformDataWrite()is used automatically more often, this uncovers further parallel flushing bugs. So, these two items are treated together in this PR.In a follow-up PR later on, as a more breaking change, we would also flush all open iterations in MPI-parallel contexts on
series.flush(), but for this we will first need functionality to reopen iterations after close #1592.TODO: