#2074: support sparse OfflineLB maps #2145

thearusable · 2023-05-09T16:12:08Z

Closes #2074

~~Depends on #2137~~
~~Depends on #2250~~

github-actions · 2023-05-09T17:03:59Z

Pipelines results

PR tests (gcc-12, ubuntu, mpich)

Build for 30cc7f4 (2024-04-22 15:38:34 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-9, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-9, ubuntu, mpich, zoltan, json schema test)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-12, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-11, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-13, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-14, ubuntu, mpich)

Build for 30cc7f4 (2024-04-22 15:38:34 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-8, ubuntu, mpich, address sanitizer)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-11, ubuntu, mpich, trace runtime, coverage)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-10, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (intel icpc, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get %0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

PR tests (nvidia cuda 11.2, gcc-9, ubuntu, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]"
          detected during:
            instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]" 
/vt/src/vt/objgroup/proxy/proxy_objgroup.impl.h(204): here
            instantiation of "vt::objgroup::proxy::Proxy<ObjT>::PendingSendType vt::objgroup::proxy::Proxy<ObjT>::reduce<f,Op,Target,Args...>(Target, Args &&...) const [with ObjT=vt::vrt::collection::lb::GreedyLB, f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Op=vt::collective::PlusOp, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>, Args=<vt::vrt::collection::lb::GreedyPayload>]" 
/vt/src/vt/vrt/collection/balance/greedylb/greedylb.cc(222): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

PR tests (clang-13, alpine, mpich)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (intel icpx, ubuntu, mpich)

Build for 30cc7f4 (2024-04-22 15:38:34 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (nvidia cuda 12.2.0, gcc-9, ubuntu, mpich)

Build for 30cc7f4 (2024-04-22 15:38:34 UTC)

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "double" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, double> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "int" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, int> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Testing - passed

Build log

PR tests (intel icpx, ubuntu, mpich, verbose)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-12, ubuntu, mpich, verbose)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-14, ubuntu, mpich, verbose)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (nvidia cuda 12.2.0, gcc-9, ubuntu, mpich, verbose)

Build for 2bc70b0 (2024-06-04 15:49:10 UTC)

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "double" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, double> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "int" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, int> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

/vt/tests/perf/send_cost.cc(169): warning #177-D: variable "prevNode" was declared but never referenced
    auto const prevNode = (thisNode - 1 + num_nodes_) % num_nodes_;
               ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

Testing - passed

Build log

tests/unit/lb/test_lb_reader.nompi.cc

tests/unit/lb/test_offlinelb.cc

nlslatt · 2023-05-23T20:44:33Z

src/vt/vrt/collection/balance/offlinelb/offlinelb.cc

 void OfflineLB::runLB(TimeType) {
-  auto const& distro = theLBDataReader()->getDistro(phase_ + 1);
-  for (auto&& elm : distro) {
+  auto nextPhase = theLBDataReader()->findNextPhase(phase_);
+  auto const distro = theLBDataReader()->getDistro(nextPhase);
+  for (auto&& elm : *distro) {
    migrateObjectTo(elm, theContext()->getNode());
  }
-  theLBDataReader()->clearDistro(phase_ + 1);
+  theLBDataReader()->clearDistro(nextPhase);
 }


If phase_ + 1 was skipped, does this new logic grab phase_ + 2 if it exists (whether explicitly described in the file or listed as identical) as a surrogate?

@lifflander Is this the desired behavior or do we want to disallow running OfflineLB when we're missing information about the phase?

Yes exactly, it will grab phase_ + 2 or whatever is the next existing phase.

@nlslatt I changed the behavior to disallow running of OfflineLB when the phase is skipped.

nlslatt · 2023-05-23T20:46:41Z

src/vt/vrt/collection/balance/lb_data_restart_reader.cc

-      if (local_changed_distro[i]) {
+    PhaseType curr = 0, next;
+    for (;curr < num_phases_ - 1;) {
+      next = findNextPhase(curr);


Similar to my other question, do we want to allow OfflineLB to operate over data from non-consecutive phases (where one is explicitly listed as skipped as opposed to just identical)?

@nlslatt I updated that to operate only on consecutive phases.

thearusable · 2023-10-19T14:01:24Z

@nlslatt, @lifflander - I rebased this PR and fixed all conflicts. PR is ready for review.

nlslatt

The testing still needs work.

tests/unit/lb/test_lb_reader.nompi.cc

tests/unit/lb/test_offlinelb.cc

nlslatt · 2023-11-06T18:17:05Z

tests/unit/lb/test_offlinelb.cc

+    "0 OfflineLB\n"
+    "1 NoLB\n"
+    "2 NoLB\n"
+    "3 NoLB\n"
+    "4 OfflineLB\n"
+    "5 OfflineLB\n"
+    "6 NoLB\n";


I think this configuration, coupled with your choice of which phases are identical to previous, means that OfflineLB by definition has nothing to migrate (for example, phase 1 is said to be identical to phase 0, so nothing should migrate when OfflineLB runs on phase 0). As such, this does not appear to be a good test of the sparse functionality.

Also, I think the phases that you have defined as identical to previous contradict what was loaded into the LBDataHolder (e.g., phase 1 is marked as identical to phase 0, but the LBDataHolder content indicates that they are not identical).

Ok, I updated the configuration for this test so the OfflineLB will be able to do some migrations. Also I removed unnecessary phases from LBDataHolder.
If you have more suggestions, please let me know.

src/vt/vrt/collection/balance/lb_data_restart_reader.h

…ig file

…for OfflineLB

…meters

…fflineLB is not configured properly

thearusable · 2024-06-04T15:50:28Z

@nlslatt Ok, I changed it to abort when VT is misconfigured.

nlslatt · 2024-06-04T16:23:03Z

@nlslatt Ok, I changed it to abort when VT is misconfigured.

@thearusable Your check only covers one of two mechanisms for enabling OfflineLB. Perhaps first check if OfflineLB is enabled by any mechanism, and then abort if the necessary flag was not passed.

thearusable · 2024-06-04T16:26:53Z

@nlslatt Ah, yes, right! I'll update that

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from 744a987 to 09b0807 Compare May 9, 2023 16:26

nmm0 mentioned this pull request May 9, 2023

Meeting Agenda [do not close] #925

Open

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch 2 times, most recently from 3fa3511 to 5b75e31 Compare May 16, 2023 16:33

thearusable self-assigned this May 16, 2023

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from 5b75e31 to f9c5f29 Compare May 22, 2023 10:07

thearusable requested review from PhilMiller, stmcgovern, lifflander, cz4rs, nmm0, JacobDomagala and nlslatt May 22, 2023 16:39

thearusable marked this pull request as ready for review May 22, 2023 16:39

nlslatt reviewed May 23, 2023

View reviewed changes

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from 0d2eb73 to 044d7c4 Compare June 5, 2023 11:13

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch 2 times, most recently from ef91bad to 8b20790 Compare June 12, 2023 14:56

thearusable requested a review from nlslatt June 13, 2023 11:56

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from 56592af to c9120ba Compare June 20, 2023 17:08

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from c9120ba to 3f0a613 Compare October 19, 2023 14:00

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from 3f0a613 to b3e07a4 Compare October 31, 2023 14:50

nlslatt requested changes Nov 6, 2023

View reviewed changes

thearusable marked this pull request as draft November 29, 2023 13:54

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch 2 times, most recently from 8a72a9f to 77911e8 Compare November 30, 2023 18:05

thearusable requested a review from nlslatt November 30, 2023 18:17

thearusable marked this pull request as ready for review November 30, 2023 18:17

thearusable added 26 commits June 4, 2024 17:49

#2074: Allow for sparse maps in LBDataRestartReader

ea129f6

#2074: Update finding next phase for OfflineLB

e6702df

#2074: Create LBDataRestartReader when there is OfflineLB in the conf…

5451f97

…ig file

#2074: Avoid adding data for identical phases to history

e2c1826

#2074: Add tests for detecting OfflineLB in LB data file

75a88d4

#2074: Add checks in sparse handler

f499358

#2074: Disallow runing of OfflineLB for skipped phases.

10dee3e

#2074: Disallow operating on non-consecutive phases

6c5e1d9

#2074: Update implementation after resolving conflicts

d45ca61

#2074: Use getUniqueFilename for file name in offlinelb tests

d889d85

#2074: Update data selection for offlineLB sparse test

e4bb6a5

#2074: Calculate number of phases after removal of the count_ property

98857f6

#2074: Fix logic for exsiting offlineLB test

8cec719

#2074: Add case to the offlinelb test which will result in assert

97705c3

#2074: Update sparseHandler in offlineLB tests

4b9a6e2

#2074: Remove failing part of the test

31a1190

#2074: Check the phase if needs OfflineLB even if requested in config

537b605

#2074: Fix crash when using param: --vt_lb_name=OfflineLB

a4fd7dc

#2074: Add unit tests to cover different initialization combinations …

b8112de

…for OfflineLB

#2074: Enable new tests only when Load Balancing is enabled

5b61a68

#2074: Add flag for gcc-8 to compile std::filesystem

bdb691b

#2074: Fix creation of the test files

47c40f2

#2074: Update unit tests to cover more combinations of OfflineLB para…

cfbdf9e

…meters

#2074: Allow to create LBDataRestartReader when lb_data_in is not used

4f9792d

#2074: Abort when OfflineLB is misconfigured

a6ef80f

#2074: Update tests after the change to show the error message when O…

2bc70b0

…fflineLB is not configured properly

thearusable force-pushed the 2074-support-sparse-offlinelb-maps branch from cf157d9 to 2bc70b0 Compare June 4, 2024 15:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2074: support sparse OfflineLB maps #2145

#2074: support sparse OfflineLB maps #2145

thearusable commented May 9, 2023 •

edited

github-actions bot commented May 9, 2023 •

edited

nlslatt May 23, 2023

thearusable Jun 5, 2023

thearusable Jun 12, 2023

nlslatt May 23, 2023

thearusable Jun 13, 2023

thearusable commented Oct 19, 2023

nlslatt left a comment

nlslatt Nov 6, 2023

thearusable Nov 30, 2023

thearusable commented Jun 4, 2024

nlslatt commented Jun 4, 2024

thearusable commented Jun 4, 2024

#2074: support sparse OfflineLB maps #2145

Are you sure you want to change the base?

#2074: support sparse OfflineLB maps #2145

Conversation

thearusable commented May 9, 2023 • edited

github-actions bot commented May 9, 2023 • edited

Pipelines results

nlslatt May 23, 2023

Choose a reason for hiding this comment

thearusable Jun 5, 2023

Choose a reason for hiding this comment

thearusable Jun 12, 2023

Choose a reason for hiding this comment

nlslatt May 23, 2023

Choose a reason for hiding this comment

thearusable Jun 13, 2023

Choose a reason for hiding this comment

thearusable commented Oct 19, 2023

nlslatt left a comment

Choose a reason for hiding this comment

nlslatt Nov 6, 2023

Choose a reason for hiding this comment

thearusable Nov 30, 2023

Choose a reason for hiding this comment

thearusable commented Jun 4, 2024

nlslatt commented Jun 4, 2024

thearusable commented Jun 4, 2024

thearusable commented May 9, 2023 •

edited

github-actions bot commented May 9, 2023 •

edited