Particle-grid periodic communication #263

streeve · 2020-06-26T19:08:10Z

This adds the first direct particle-grid capability in Cabana within core, with dependency on Cajita. ~~This is all in particle_grid/ for now, but I'm open to any reorganization and renaming. CabanaPG directly relies on Core and Cajita and is enabled by default if Cajita is.~~

This wraps calls to Distributor, migrate, Halo, and gather to simplify things for any user with periodic boundaries or load balancing needs.

New options include:

gridDistributor (stolen from ExaMPM), determining new ranks destinations and wrapping through periodic boundaries
gridMigrate using gridDistributor and calling migrate
migrate tests
gridHalo determining particles to be ghosted and any necessary periodic shifts
~~PeriodicHalo inheriting from Halo with functor to add periodic shifts~~
PeriodicShift functor to modify buffer (cannot inherit from Halo because CommPlan contains std member variables, an issue for NVCC)
gridGather using PeriodicShift and gridHalo and calling gather
Update to gather for modifying the send buffer with periodic shifts
halo tests

Closes #77

sslattery

We should discuss but my thoughts are that this should be in core. We should also discuss whether or not to include the halo region at this point in the communication. I think we could have one general cartesian communication structure (both for halo and migration) which incorporates the halo information if needed. We could do this now or treat this as a good start and do more PRs to generalize it.

particle_grid/src/Cabana_Periodic.hpp

CMakeLists.txt

particle_grid/src/Cabana_Periodic.hpp

sslattery · 2020-06-26T20:00:39Z

particle_grid/unit_test/tstPeriodic.hpp

+
+    // Create a  grid local_grid.
+    auto local_grid = Cajita::createLocalGrid( global_grid, 1 );
+    auto local_mesh = Cajita::createLocalMesh<TEST_DEVICE>( *local_grid );


Does it seem reasonable to have to make all of these objects for cases like MD with short-range only where you don't need a mesh? I could see automating how often migrate is called based on setting some halo in MD. That would then eliminate the need to tune how often migrate happens (e.g. every 100 time steps or something). It would instead be based on some distance criterion which would correspond to a halo width. In that case you would want to create all of these objects then as your cell size would have some distance relative to something meaningful in the simulation (like the interaction distance).

Note this extra grid we would use to choose communications would also be the same grid from which we would build LinkedCellList objects.

And now that I think about it we need a LinkedCellList that uses Cajita instead of the grid we have as an implementation detail. Even more reason for all of this stuff to just be in core.

I guess I'm not sure how to get around creating all these objects, but I have considered specializations of the mesh/grid that don't hold as much information since we basically only need the bounds of the box in MD

And since we have to rebuild the neighbor list every so often, we do the same with migration, but you're right that it could potentially be done automatically with a check for anything crossing halo boundaries.

I have the LinkedCellList on my list after the PeriodicHalo

streeve · 2020-06-26T23:05:23Z

I think I addressed everything except for how to potentially generalize for other particle-grid methods. Now that I did it as you suggested, this makes much more sense than a separate particle-grid section

junghans · 2020-09-03T19:48:49Z

@streeve that helped with the build, but now the Cabana_ParticleGridCommunication_test_OPENMP_1 test fails.

streeve · 2020-09-03T20:01:46Z

@streeve that helped with the build, but now the Cabana_ParticleGridCommunication_test_OPENMP_1 test fails.

Thanks - that's expected at the moment. gridMigrate passes everything, but gridGather has some more issues to fix.

streeve · 2020-09-11T23:58:37Z

This will fail until rebased with 295 in master; after that, probably a few more fixes to pass halo tests and it will be ready for review.

sslattery · 2020-09-22T13:38:55Z

#295 has been merged - let's proceed with this one and see if it resolves our comm seg faults.

sslattery · 2020-09-30T12:14:23Z

Let's rebase this one to get the latest changes for the number of testing ranks and see what the CI does. We likely still have a UVM error though.

codecov · 2020-12-18T01:31:55Z

Codecov Report

Merging #263 (0a67b77) into master (d46804f) will decrease coverage by 0.2%.
The diff coverage is 94.9%.

@@           Coverage Diff            @@
##           master    #263     +/-   ##
========================================
- Coverage    93.5%   93.3%   -0.3%     
========================================
  Files          37      38      +1     
  Lines        2797    2842     +45     
========================================
+ Hits         2617    2653     +36     
- Misses        180     189      +9

Flag	Coverage Δ
clang	`93.3% <94.9%> (-0.3%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
core/src/Cabana_CommunicationPlan.hpp	`0.0% <0.0%> (-98.3%)`	⬇️
core/src/Cabana_Halo.hpp	`96.1% <95.6%> (+0.3%)`	⬆️
core/src/Cabana_ParticleGridCommunication.hpp	`99.1% <99.1%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d46804f...afb82ac. Read the comment docs.

streeve · 2021-01-04T22:21:27Z

@sslattery this is ready for another review. @junghans codecov is telling me I reduced coverage for CommPlan to zero - any ideas?

core/src/Cabana_ParticleGridCommunication.hpp

sslattery · 2021-01-07T16:29:17Z

core/src/Cabana_ParticleGridCommunication.hpp

+*/
+template <class LocalGridType, class PositionSliceType>
+Distributor<typename PositionSliceType::device_type>
+gridDistributor( const LocalGridType& local_grid, PositionSliceType& positions )


I would prefer this be named createGridDistributor to indicate the creation mechanism.

sslattery · 2021-01-07T19:47:24Z

core/src/Cabana_ParticleGridCommunication.hpp

+*/
+template <class LocalGridType, class PositionSliceType,
+          std::size_t PositionIndex>
+auto gridHalo(


These should also be named createGridHalo

sslattery · 2021-01-07T19:50:47Z

core/src/Cabana_ParticleGridCommunication.hpp

+    // Create the Shifts.
+    auto periodic_shift = PeriodicShift<device_type, PositionIndex>( shifts );
+
+    return std::make_pair( halo, periodic_shift );


Instead of a pair we should just have a new struct which holds a Halo and the functor the user wants to apply. These would eliminate a user needed to store multiple objects and think about using the pair. In addition, it would allow the user to have an object they could easily append other modification functions if they wanted to for some reason.

Done - although I'm not sure if it's composed to be extensible in the way you mentioned

sslattery · 2021-01-07T22:21:31Z

core/unit_test/tstParticleGridCommunication.hpp

+    auto center_z = width_z + lo_z;
+    auto shift_x = width_x - ( halo_width - 0.1 ) * dx;
+    auto shift_y = width_y - ( halo_width - 0.1 ) * dy;
+    auto shift_z = width_z - ( halo_width - 0.1 ) * dz;


All of the setup code looks the same for both tests - perhaps it can be combined.

sslattery · 2021-01-07T22:23:05Z

core/unit_test/tstParticleGridCommunication.hpp

+    Cabana::AoSoA<DataTypes, TEST_MEMSPACE> data_src( "data_src", num_data );
+    Cabana::deep_copy( data_src, data_host );
+
+    if ( test_type == 0 )


Did you anticipate adding other values for test_type?

I added a slice variant of gridGather for a second test_type

sslattery · 2021-01-07T22:24:13Z

core/unit_test/tstParticleGridCommunication.hpp

+        bool within_y = true;
+        bool within_z = true;
+        // Make sure all ghosts are in halo region in at least one direction.
+        if ( pos_host( i, Cajita::Dim::I ) < lo_x )


Could you add a particle field which contains the original MPI rank and then use that to back-calculate what the position should be exactly?

streeve added the enhancement New feature or request label Jun 26, 2020

streeve requested a review from sslattery June 26, 2020 19:08

sslattery assigned streeve Jun 26, 2020

sslattery requested changes Jun 26, 2020

View reviewed changes

streeve force-pushed the periodic_migrate branch from fea92a3 to 7fe48af Compare June 26, 2020 22:37

streeve marked this pull request as draft July 2, 2020 19:57

streeve changed the title ~~Particle-grid periodic migrate~~ Particle-grid periodic communication Jul 2, 2020

streeve force-pushed the periodic_migrate branch 2 times, most recently from 64db1ca to fc4eb4e Compare July 9, 2020 19:32

streeve force-pushed the periodic_migrate branch from fc4eb4e to a3c8aef Compare July 16, 2020 20:38

streeve changed the title ~~Particle-grid periodic communication~~ WIP: Particle-grid periodic communication Jul 16, 2020

streeve force-pushed the periodic_migrate branch 2 times, most recently from 73e13cb to 73966be Compare August 6, 2020 03:13

streeve force-pushed the periodic_migrate branch 2 times, most recently from 6efa3c4 to e724c72 Compare August 8, 2020 00:14

streeve force-pushed the periodic_migrate branch from e724c72 to dd5cfd5 Compare September 3, 2020 19:04

streeve mentioned this pull request Sep 11, 2020

CommunicationPlan copy #295

Merged

streeve force-pushed the periodic_migrate branch from 95a61b3 to d519cc0 Compare September 11, 2020 23:54

streeve force-pushed the periodic_migrate branch from e322c43 to b24a18c Compare September 24, 2020 00:12

streeve marked this pull request as ready for review September 24, 2020 00:17

streeve closed this Sep 24, 2020

streeve reopened this Sep 24, 2020

streeve force-pushed the periodic_migrate branch from 7164526 to 0cd812e Compare September 30, 2020 19:34

streeve force-pushed the periodic_migrate branch from 0cd812e to 271b352 Compare November 19, 2020 17:35

streeve force-pushed the periodic_migrate branch from 271b352 to 89e8276 Compare November 19, 2020 17:55

streeve force-pushed the periodic_migrate branch from 89e8276 to 9a59977 Compare December 17, 2020 19:30

streeve changed the title ~~WIP: Particle-grid periodic communication~~ Particle-grid periodic communication Dec 17, 2020

streeve requested a review from sslattery December 17, 2020 21:33

streeve force-pushed the periodic_migrate branch 2 times, most recently from 6b0827e to 05f3ca3 Compare December 30, 2020 19:29

Add particle-grid periodic migrate and gather

7f5b0e4

streeve force-pushed the periodic_migrate branch from 05f3ca3 to 7f5b0e4 Compare January 4, 2021 17:00

sslattery reviewed Jan 7, 2021

View reviewed changes

core/src/Cabana_ParticleGridCommunication.hpp Outdated Show resolved Hide resolved

Fuse migrate periodic shift

0a67b77

sslattery requested changes Jan 7, 2021

View reviewed changes

streeve added 2 commits January 8, 2021 10:31

Address review naming and return GridHalo instead of pair

f0cd924

Share PG communication unit test data setup

552abb0

streeve force-pushed the periodic_migrate branch from 78b287e to 47fe0a8 Compare January 13, 2021 21:33

Add grid gather with slices

afb82ac

streeve force-pushed the periodic_migrate branch from 47fe0a8 to afb82ac Compare January 14, 2021 05:46

streeve mentioned this pull request May 20, 2021

Particle-grid migrate #395

Merged

3 tasks

streeve added this to In progress in New features Jan 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Particle-grid periodic communication #263

Particle-grid periodic communication #263

streeve commented Jun 26, 2020 •

edited

sslattery left a comment

sslattery Jun 26, 2020

sslattery Jun 26, 2020

sslattery Jun 26, 2020

streeve Jun 26, 2020

streeve commented Jun 26, 2020 •

edited

junghans commented Sep 3, 2020

streeve commented Sep 3, 2020

streeve commented Sep 11, 2020

sslattery commented Sep 22, 2020

sslattery commented Sep 30, 2020

codecov bot commented Dec 18, 2020 •

edited

streeve commented Jan 4, 2021

sslattery Jan 7, 2021

streeve Jan 13, 2021

sslattery Jan 7, 2021

streeve Jan 13, 2021

sslattery Jan 7, 2021

streeve Jan 13, 2021

sslattery Jan 7, 2021

streeve Jan 13, 2021

sslattery Jan 7, 2021

streeve Jan 13, 2021

sslattery Jan 7, 2021

Particle-grid periodic communication #263

Are you sure you want to change the base?

Particle-grid periodic communication #263

Conversation

streeve commented Jun 26, 2020 • edited

sslattery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

streeve commented Jun 26, 2020 • edited

junghans commented Sep 3, 2020

streeve commented Sep 3, 2020

streeve commented Sep 11, 2020

sslattery commented Sep 22, 2020

sslattery commented Sep 30, 2020

codecov bot commented Dec 18, 2020 • edited

Codecov Report

streeve commented Jan 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

streeve commented Jun 26, 2020 •

edited

streeve commented Jun 26, 2020 •

edited

codecov bot commented Dec 18, 2020 •

edited