[WIP] OCCA Backend Update #1043

kris-rowe · 2022-08-19T22:09:58Z

When complete, this PR will update the OCCA backend to be compatible with OCCA v1.4 and add support for the OCCA OpenCL and SYCL backends.

This PR will close #816.
This PR conflicts with #1007.

jedbrown · 2022-08-19T22:14:32Z

Thanks @kris-rowe! Feel free to tag in myself or @jeremylt at any time.

jedbrown · 2022-08-22T21:02:22Z

Great, I see this compiling. This issue is probably an easy fix, but I get this failure.

$ make test search=t001 BACKENDS=/cpu/self/occa
[...]
not ok 3 t001-ceed /cpu/self/occa stderr
# +/home/jed/src/libCEED/backends/occa/ceed-occa.cpp:336 in registerBackend():
# +---[ Error ]--------------------------------------------------------------------
# +File     : /home/jed/src/occa/src/types/json.cpp
# +Line     : 491
# +Function : operator[]
# +Message  : Path '' is not an object
# +Stack
# +4 build/t001-ceed(+0x11a6)
# +3 /usr/lib/libc.so.6(+0x232d0)
# +2 /usr/lib/libc.so.6(__libc_start_main+0x8a)
# +1 build/t001-ceed(+0x10a5)

Building with OPT='-g' gives this trace

#0  0x00007ffff776a4dc in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff771a998 in raise () from /usr/lib/libc.so.6
#2  0x00007ffff770453d in abort () from /usr/lib/libc.so.6
#3  0x00007ffff7a32887 in CeedErrorAbort (ceed=0x5555558af080, filename=0x7ffff7bb36b8 "/home/jed/src/libCEED/backends/occa/ceed-occa.cpp", line_no=336, func=0x7ffff7bb37c9 "registerBackend", err_code=-2, format=0x5555558b4010 "\n---[ Error ]", '-' <repeats 68 times>, "\n    File     : /home/jed/src/occa/src/types/json.cpp\n    Line     : 491\n    Function : operator[]\n    Message  : Path "..., args=0x7fffffffd060) at /home/jed/src/libCEED/interface/ceed.c:1245
#4  0x00007ffff7a325c5 in CeedErrorImpl (ceed=0x5555558af080, filename=0x7ffff7bb36b8 "/home/jed/src/libCEED/backends/occa/ceed-occa.cpp", lineno=336, func=0x7ffff7bb37c9 "registerBackend", ecode=-2, format=0x5555558b4010 "\n---[ Error ]", '-' <repeats 68 times>, "\n    File     : /home/jed/src/occa/src/types/json.cpp\n    Line     : 491\n    Function : operator[]\n    Message  : Path "...) at /home/jed/src/libCEED/interface/ceed.c:1168
#5  0x00007ffff7aa3b42 in ceed::occa::registerBackend (resource=0x7fffffffde0b "/cpu/self/occa", ceed=0x5555558af080) at /home/jed/src/libCEED/backends/occa/ceed-occa.cpp:336
#6  0x00007ffff7a31e0a in CeedInit (resource=0x7fffffffde0b "/cpu/self/occa", ceed=0x7fffffffd7e0) at /home/jed/src/libCEED/interface/ceed.c:967
#7  0x00005555555551b8 in main (argc=2, argv=0x7fffffffd908) at /home/jed/src/libCEED/tests/t001-ceed.c:10

kris-rowe · 2022-08-22T22:05:28Z

I tracked down the configuration issue. Will see if the rest of the tests pass locally now.

jeremylt · 2022-08-24T23:13:47Z

backends/occa/include/ceed.h

@@ -0,0 +1,4 @@
+#ifndef _OCCA_INCLUDE_CEED_H_


If you want to include this during JiT, see the example files like this in `/include/ceed/jit-source/cuda/cuda-jit.h'

Any files that will be used as JiT source should go in include/ceed/jit-source/occa

jeremylt · 2022-08-24T23:16:58Z

include/ceed/jit-source/gallery/ceed-mass2dbuild.h

@@ -19,10 +19,11 @@ CEED_QFUNCTION(Mass2DBuild)(void *ctx, const CeedInt Q,
  // *INDENT-OFF*
  // in[0] is Jacobians with shape [2, nc=2, Q]
  // in[1] is quadrature weights, size (Q)
-  const CeedScalar (*J)[2][CEED_Q_VLA] = (const CeedScalar(*)[2][CEED_Q_VLA])in[0],
-                                    *w = in[1];
+  typedef CeedScalar array_t[3][CEED_Q_VLA];


I don't think that we want to force users to have to do this.

For sure. This is a temporary hack since there is a deadline next week.

I will update the OCCA parser to handle pointers-to-arrays correctly in the near future.

jedbrown · 2022-09-06T02:45:14Z

Our CI is back online so I think we'll be able to test this now with CUDA and/or ROCm. We have oneAPI compilers installed, though not with Intel hardware. What would you consider a good choice for testing?

jedbrown · 2022-09-07T15:59:21Z

examples/fluids/qfunctions/channel.h

@@ -29,11 +28,12 @@ struct ChannelContext_ {
  CeedScalar B;        // !< Body-force driving the flow
  struct NewtonianIdealGasContext_ newtonian_ctx;
 };
+#define ChannelContext struct ChannelContext_*;


Suggested change

#define ChannelContext struct ChannelContext_*;

#define ChannelContext struct ChannelContext_*

jedbrown · 2022-09-07T16:02:53Z

I can run this variant of the channel test (which includes output validation) with /cpu/self/occa and /gpu/cuda, but it fails with /gpu/cuda/occa.

$ build/fluids-navierstokes -options_file examples/fluids/channel.yaml -compare_final_state_atol 2e-11 -compare_final_state_filename examples/fluids/tests-output/fluids-navierstokes-channel.bin -dm_plex_box_faces 5,5,1 -ts_max_steps 5 -ceed /cpu/self/occa -dm_mat_preallocate_skip 0 -snes_fd_color

The trace is:

Thread 1 "fluids-navierst" received signal SIGSEGV, Segmentation fault.
0x00007fffbae86c51 in occa::modeMemory_t::addMemoryRef(occa::memory*) () from /home/jed/src/occa/lib/libocca.so
(gdb) bt
#0  0x00007fffbae86c51 in occa::modeMemory_t::addMemoryRef(occa::memory*) () from /home/jed/src/occa/lib/libocca.so
#1  0x00007ffff5aeed56 in ceed::occa::arrayToMemory<double> (array=array@entry=0x7ffe4be05400) at /home/jed/src/libCEED/backends/occa/ceed-occa-vector.hpp:27
#2  0x00007ffff5aecdef in ceed::occa::Vector::useArrayPointer (this=0x555559006800, mtype=<optimized out>, array=0x7ffe4be05400) at /home/jed/src/libCEED/backends/occa/ceed-occa-vector.cpp:246
#3  0x00007ffff5aed2f0 in ceed::occa::Vector::setArray (this=0x555559006800, mtype=mtype@entry=CEED_MEM_DEVICE, cmode=cmode@entry=CEED_USE_POINTER, array=array@entry=0x7ffe4be05400) at /home/jed/src/libCEED/backends/occa/ceed-occa-vector.cpp:166
#4  0x00007ffff5aed366 in ceed::occa::Vector::ceedSetArray (vec=<optimized out>, mtype=CEED_MEM_DEVICE, cmode=CEED_USE_POINTER, array=0x7ffe4be05400) at /home/jed/src/libCEED/backends/occa/ceed-occa-vector.cpp:450
#5  0x00007ffff5aac1fe in CeedVectorSetArray (vec=0x5555590f9920, mem_type=<optimized out>, copy_mode=copy_mode@entry=CEED_USE_POINTER, array=<optimized out>) at /home/jed/src/libCEED/interface/ceed-vector.c:274
#6  0x000055555557d9ea in ICs_FixMultiplicity (dm=0x555555d65480, ceed_data=0x555555d11720, user=<optimized out>, Q_loc=0x55555911d3b0, Q=0x555559118b30, time=<optimized out>, time@entry=0) at /home/jed/src/libCEED/examples/fluids/src/misc.c:38
#7  0x000055555555a4ef in main (argc=<optimized out>, argv=<optimized out>) at /home/jed/src/libCEED/examples/fluids/navierstokes.c:174

I'm just testing locally so far; does this work for you at JLSE?

kris-rowe · 2022-09-08T16:34:12Z

Our CI is back online so I think we'll be able to test this now with CUDA and/or ROCm. We have oneAPI compilers installed, though not with Intel hardware. What would you consider a good choice for testing?

The OCCA CUDA backend should be sufficient for testing. If you really wanted to test the SYCL backend specifically, you could build the public Intel LLVM compilers with the SYCL CUDA plugin enabled.

jeremylt · 2022-09-26T15:47:36Z

Correct me if I'm wrong, but I understand from the comments and commits that the core tests, t1*-t5* and ex1, ex2 all pass with these changes.

In preparation for our upcoming release, I'd like to get this initial work merged. Specifically, I would like to merge the changes in Makefile, backends/occa/*, backends/ceed-backend-list.h and OMIT the changes in examples/*, tests/*, and include/ceed/jit-source/gallery/*

I think a couple of changes need to be added

Update for changes in main (rebase or merge?)
Move files used in JiT to include/ceed/jit-source/occa
Update tests/junit.py (replaces tests/tap.sh to only run tests for t1*-t3* on OCCA backends
Update README.md with new OCCA min version and backends
Update docs/sphinx/source/releasenotes.md

I can help with any/all of those changes

jeremylt · 2022-10-11T15:04:52Z

Thanks for all the hard work here @kris-rowe!

PETSc released last week, so I'd like to tidy up the big open PRs and get a libCEED release soon. I don't want to step on any toes, but I have the time to run down those small tasks I listed above so we get this PR into the release if that's ok with you.

kris-rowe · 2022-10-11T15:08:35Z

PETSc released last week, so I'd like to tidy up the big open PRs and get a libCEED release soon. I don't want to step on any toes, but I have the time to run down those small tasks I listed above so we get this PR into the release if that's ok with you.

If you have time that would be a huge help.

jeremylt · 2022-10-12T16:10:06Z

Ok, #1072 should be ready to merge now

jedbrown · 2022-10-13T20:08:39Z

Shall we rebase this after #1072? And is it possible to make block diagonal assembly fall back to CPU for now? This will allow us to run ceed-fluids on PVC with a narrowly localized performance distortion.

jeremylt · 2022-10-25T19:31:46Z

backends/occa/include/ceed.h

+#ifndef _OCCA_INCLUDE_CEED_H_
+#define _OCCA_INCLUDE_CEED_H_
+// Phony header to include when compiling OKL
+#endif


It's really intended that JiT headers don't go here in the backend folder.

jedbrown · 2022-12-02T16:12:15Z

examples/fluids/qfunctions/setupgeo2d.h

+<<<<<<< HEAD
  const CeedScalar(*J)[CEED_Q_VLA] = (const CeedScalar(*)[CEED_Q_VLA])in[0];
  const CeedScalar(*w)             = in[1];

  // Outputs
  CeedScalar(*q_data_sur)[CEED_Q_VLA] = (CeedScalar(*)[CEED_Q_VLA])out[0];
+=======
+  typedef CeedScalar vec_t[CEED_Q_VLA];
+  const vec_t* J = (const vec_t*) in[0];
+  const CeedScalar * const w = in[1];
+  // Outputs
+  vec_t * q_data_sur = (vec_t*) out[0];
+  // *INDENT-ON*
+>>>>>>> f0e9d76d (Rewrites fluids example qfunctions to be compatible with OCCA.)


Conflict markers

jedbrown · 2022-12-02T16:22:23Z

Here's a suggested test.

diff --git i/examples/fluids/blasius.yaml w/examples/fluids/blasius.yaml
index cf3056b1e..fd7516ee6 100644
--- i/examples/fluids/blasius.yaml
+++ w/examples/fluids/blasius.yaml
@@ -4,7 +4,7 @@ implicit: true
 ts:
   adapt_type: 'none'
   type: 'beuler'
-  dt: 0.2e-5
+  dt: 0.1e-5
   max_time: 1.0e-3
 output_freq: 10
 
@@ -51,3 +51,15 @@ stg:
   use: false
   inflow_path: "./STGInflow_blasius.dat"
   mean_only: true
+
+pmat_pbdiagonal:
+ksp_type: bcgsl
+pc_type: vpbjacobi
+amat_type: shell
+
+# monitors
+ts_monitor:
+snes_monitor:
+ksp_converged_reason:
+
+ceed: /gpu/cuda

It should produce output like

 build/fluids-navierstokes  -options_file examples/fluids/blasius.yaml
[...]
0 TS dt 1e-06 time 0.
    0 SNES Function norm 6.801978174922e-04
    Linear solve converged due to CONVERGED_RTOL iterations 222
    1 SNES Function norm 1.515152456321e-07
    Linear solve converged due to CONVERGED_RTOL iterations 286
    2 SNES Function norm 4.628934945817e-11
1 TS dt 1e-06 time 1e-06
[...]

jedbrown · 2022-12-02T16:34:09Z

For performance monitoring, you can add -log_view -log_view_gpu_time.

jeremylt reviewed Aug 24, 2022

View reviewed changes

jedbrown added this to the v0.11 milestone Sep 6, 2022

jedbrown mentioned this pull request Sep 6, 2022

Additional backends #26

Closed

6 tasks

jedbrown reviewed Sep 7, 2022

View reviewed changes

jeremylt mentioned this pull request Oct 13, 2022

OCCA Parser Handling Pointers to Arrays libocca/occa#628

Open

jeremylt removed this from the v0.11 milestone Oct 17, 2022

kris-rowe force-pushed the occa-backend-update branch from 8801fe3 to d15d972 Compare October 25, 2022 19:24

jeremylt reviewed Oct 25, 2022

View reviewed changes

kris-rowe added 10 commits December 1, 2022 21:55

Removes deprecated calls to occa::linalg.

c4ab772

Adds a dummy ceed.h header in include in OCCA kernels.

31a7629

Rewrite arrays of structs in format that OCCA can handle.

1a3a1f9

Includes the cmath header when compiling C++ code.

3df7633

Removes math function headers which were causing OCCA JIT failures.

f1b142f

Rewrite arrays of structs in format that OCCA can handle.

1a6fffd

Rewrites fluids example qfunctions to be compatible with OCCA.

0aacae8

Rewrites advection problem kernels to work with OCCA.

a879016

Rewrites blasius problem kernels to work with OCCA.

177848b

Rewrites channel problem kernels to work with OCCA.

4b11b94

kris-rowe added 8 commits December 1, 2022 22:16

Rewrites dirichlet bc kernels to work with OCCA.

46d6695

Rewrites newtonian kernels to work with OCCA.

aed20a0

Rewrites setupgeo kernels to be compatible with OCCA.

818ef30

Rewrites stabilization kernels to be compatible with OCCA.

b97eee4

Rewrites stg kernels to be compatible with OCCA.

9890d73

Saves WIP before rebasing onto upstream libCEED.

df482e8

Rewrites pointer-to-array in t507 to be compatible with OCCA.

f75f425

Removes filters excluding OCCA from certain tests.

1760b84

kris-rowe force-pushed the occa-backend-update branch from d15d972 to 1760b84 Compare December 1, 2022 22:47

jedbrown reviewed Dec 2, 2022

View reviewed changes

kris-rowe added 3 commits December 6, 2022 17:01

Removes previous merge conflicts.

eca1eaf

Includes correct math header to be compatible with OCCA.

bc3b4f6

Appends OCCA_DIR/lib to LD_LIBRARY_PATH instead of overwriting.

2d6dba8

kris-rowe mentioned this pull request Dec 6, 2022

Array declaration with initialization in OKL causes segfault during parsing. libocca/occa#640

Open

Save WIP on OCCA backend to push to GitHub.

0c5cb32

jeremylt added the 0-WIP label Jan 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] OCCA Backend Update #1043

[WIP] OCCA Backend Update #1043

kris-rowe commented Aug 19, 2022

jedbrown commented Aug 19, 2022

jedbrown commented Aug 22, 2022

kris-rowe commented Aug 22, 2022

jeremylt Aug 24, 2022

jeremylt Aug 24, 2022

kris-rowe Aug 24, 2022

jedbrown commented Sep 6, 2022

jedbrown Sep 7, 2022

jedbrown commented Sep 7, 2022

kris-rowe commented Sep 8, 2022

jeremylt commented Sep 26, 2022 •

edited

jeremylt commented Oct 11, 2022

kris-rowe commented Oct 11, 2022

jeremylt commented Oct 12, 2022

jedbrown commented Oct 13, 2022

jeremylt Oct 25, 2022 •

edited

jedbrown Dec 2, 2022

jedbrown commented Dec 2, 2022

jedbrown commented Dec 2, 2022

	#define ChannelContext struct ChannelContext_*;
	#define ChannelContext struct ChannelContext_*

[WIP] OCCA Backend Update #1043

Are you sure you want to change the base?

[WIP] OCCA Backend Update #1043

Conversation

kris-rowe commented Aug 19, 2022

jedbrown commented Aug 19, 2022

jedbrown commented Aug 22, 2022

kris-rowe commented Aug 22, 2022

jeremylt Aug 24, 2022

Choose a reason for hiding this comment

jeremylt Aug 24, 2022

Choose a reason for hiding this comment

kris-rowe Aug 24, 2022

Choose a reason for hiding this comment

jedbrown commented Sep 6, 2022

jedbrown Sep 7, 2022

Choose a reason for hiding this comment

jedbrown commented Sep 7, 2022

kris-rowe commented Sep 8, 2022

jeremylt commented Sep 26, 2022 • edited

jeremylt commented Oct 11, 2022

kris-rowe commented Oct 11, 2022

jeremylt commented Oct 12, 2022

jedbrown commented Oct 13, 2022

jeremylt Oct 25, 2022 • edited

Choose a reason for hiding this comment

jedbrown Dec 2, 2022

Choose a reason for hiding this comment

jedbrown commented Dec 2, 2022

jedbrown commented Dec 2, 2022

jeremylt commented Sep 26, 2022 •

edited

jeremylt Oct 25, 2022 •

edited