Add kernel fusing using RAJA #167

davidbeckingsale · 2021-03-04T21:24:14Z

Make CoarsenCopyTransaction inherit from FuseableTransaction
Flesh out KernelFuser
Add KernelFuser argument to loop API

Fuseable transactions still need to be fused.

MrBurmark · 2021-09-16T21:46:07Z

source/SAMRAI/tbox/ExecutionPolicy.h

@@ -111,6 +111,11 @@ struct policy_traits<policy::parallel> {
   >;

   using ReductionPolicy = RAJA::cuda_reduce;
+
+   using WorkGroupPolicy = RAJA::WorkGroupPolicy<
+      RAJA::cuda_work_async<1024>, 


Can you provide a way to set the workgroup block size in case people run into cuda linking issues?

MrBurmark · 2021-09-16T21:52:11Z

source/SAMRAI/tbox/Schedule.C

+         }
+         d_recv_fuser->launch();
+#if defined(HAVE_RAJA)
+         parallel_synchronize();


Is this synchronization between fused and non-fused necessary?

If it is necesary, what about if d_recv_sets_fuseable[sender] is empty?

It shouldn't be needed if it's empty and the above loop is a no-op. I think also the launch() call is unnecessary in that case.

MrBurmark · 2021-09-17T17:57:02Z

source/SAMRAI/hier/PatchData.h

      const PatchData& src,
      const BoxOverlap& overlap) = 0;

+   virtual void
+   copy(
+      const PatchData& src,
+      const BoxOverlap& overlap,
+      tbox::KernelFuser& fuser);


As it stands I'll have to implement both if I want to use fusion. I guess I can have one implementation for both that takes the fuser pointer and use or not use it under an abstraction layer to keep things single source. I'll need to use some macros to maintain support for older versions of samrai but that was pretty much inevitable.

MrBurmark · 2021-09-30T17:07:13Z

source/SAMRAI/hier/PatchData.h

+   packStream(
+      tbox::MessageStream& stream,
+      const BoxOverlap& overlap,
+      tbox::KernelFuser& fuser);


Yes, looks like this should be const.

MrBurmark · 2021-09-30T21:22:11Z

Could you add a feature availability macro in your config file so its easy to check if the fuser exists?

in pdat classes that take KernelFuser.

MrBurmark · 2021-11-17T19:09:14Z

config/SAMRAI_config.h.cmake.in

@@ -358,7 +358,9 @@
 /* Configure for compiling on BGL family of machines */
 #undef __BGL_FAMILY__

-
+#ifdef HAVE_RAJA
+#define HAVE_KERNEL_FUSER


SAMRAI_HAVE_KERNEL_FUSER?

MrBurmark · 2021-11-18T22:27:51Z

source/SAMRAI/tbox/Schedule.C

      }
-      TBOX_ASSERT(mi->first == send_coms[icom].getPeerRank());
+#if defined(HAVE_RAJA)      
+      parallel_synchronize();


fuser cleanup call

for_all calls

and add needed guards for non-cuda builds

reallocations

…re/fuse-comm

have been launched

StagedKernelFusers

launched kernels.

davidbeckingsale and others added 4 commits December 11, 2020 13:02

Add TransactionFuseable class

c9ac694

Fixup handling fuseable/regular transactions

5ff40e2

Fuseable transactions still need to be fused.

Adding new API for fusing

5cc28c3

Add guard for non-umpire builds

beccc9f

MrBurmark reviewed Sep 16, 2021

View reviewed changes

MrBurmark reviewed Sep 17, 2021

View reviewed changes

MrBurmark reviewed Sep 30, 2021

View reviewed changes

nselliott and others added 5 commits September 30, 2021 15:02

Add #define to signal existance of KernelFuser

cdf1111

Merge branch 'master' into feature/fuse-comm

049bfd0

Make CoarsenCopyTransaction fuseable

6d600c8

Merge branch 'master' into feature/nselliott/kernel-fuser

c0a037d

Start to turn on workgroups for KernelFuser, and add placeholder methods

91f0636

in pdat classes that take KernelFuser.

MrBurmark reviewed Nov 17, 2021

View reviewed changes

Add to the implementation of KernelFuser

31b3798

MrBurmark reviewed Nov 18, 2021

View reviewed changes

nselliott added 13 commits November 18, 2021 15:31

Add kernel fuser allocator to AllocatorDatabase

c249b86

Add checks to do fuser launch and synchronize only when needed.

e91952a

Add kernel fuser cleanup

3ab15d8

Change KernelFuser to a singleton and begin adding it to some ArrayData

eaf6fd2

for_all calls

Add enqueue calls to some of the for_alls, add fuser in more places

d43092e

Change name of virtual methods for fuseable operations in PatchData,

f426913

and add needed guards for non-cuda builds

Add missing initialization in ArrayData

f8a5518

Make KernelFuser a true no-op in non-RAJA builds.

4fe3d2c

Rearrange split message receives in AsyncCommPeer to avoid CUDA

c57914d

reallocations

Add cmake option for setting number of threads for RAJA WorkGroup policy

b6fee9f

Merge branch 'feature/fuse-comm' of github.com:LLNL/SAMRAI into featu…

d9c4ce3

…re/fuse-comm

Add logic to avoid synchronize calls when it is known that no kernels

899a9fa

have been launched

Merge branch 'master' into feature/fuse-comm-merge

9b08ece

nselliott added 15 commits April 4, 2022 16:35

Add methods for applications to indicate need for synchronization

7c24184

Clarify some documentation comments

78c618d

Add option to set a synchronize between refine and postprocessRefine

2b1a9e1

Add KernelFuserStages as a singleton to hold and use KernelFuser

1eb8cf5

Add ScheduleKernelFuser and change KernelFuserStages to

5a62605

StagedKernelFusers

Add StagedKernelFusers launch/cleanup in RefineSchedule

879e75e

Stop some synchronize calls between the communicate and refine steps

67ef461

Change tbox::Schedule to use StagedKernelFusers

8a8a98e

Add optional synchronization around boundary conditions.

f2230e9

Add StagedKernelFusers calls to PatchLevel allocate`

8df7260

Add cuda dependency in some tests

6f6c6bd

Add kernel fusion calls to PatchLevel deallocate and CoarsenSchedule

6397677

Revise StagedKernelFusers, remove isActive, add check on whether it

41a65a6

launched kernels.

Remove stray printf

cb6fa43

Small fixes to work with RAJA 2022.03

4837d89

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kernel fusing using RAJA #167

Add kernel fusing using RAJA #167

davidbeckingsale commented Mar 4, 2021 •

edited

MrBurmark Sep 16, 2021

MrBurmark Sep 16, 2021

MrBurmark Sep 16, 2021

nselliott Sep 17, 2021

MrBurmark Sep 17, 2021

MrBurmark Sep 30, 2021

nselliott Sep 30, 2021

MrBurmark commented Sep 30, 2021

MrBurmark Nov 17, 2021

MrBurmark Nov 18, 2021

Add kernel fusing using RAJA #167

Are you sure you want to change the base?

Add kernel fusing using RAJA #167

Conversation

davidbeckingsale commented Mar 4, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrBurmark commented Sep 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidbeckingsale commented Mar 4, 2021 •

edited