exceeds register size error when building with SYCL backend

[XzzX]

When using AoSoA datastructure and slices Intel compilers will exceeds register size when building with SYCL backend for Intel GPUs.

With the help of @igorvorobtsov, the problem was identified being the labels. Labels in Cabana are stored as char arrays. The Intel compiler will expand each char to 4 bytes for alignment reasons. By passing slices by value to kernels this can exceed the register size.

[XzzX]

Any progress here?

[streeve]

Sorry about the delay on this.

First, for background (briefly discussed on slack) each Cabana::slice is an unmanaged Kokkos::View for one field in the AoSoA. Since it's only pointing to existing data and nothing else unmanaged Views do not hold a label. We took the opposite approach and store char array labels in slices assuming they're useful, but we could revisit that

I haven't been able to reproduce in my most recent builds, but this sounds like it's at the application level. Do you see this in any in-repo tests or examples (and if not can you give more detail on what your kernel looks like)?

For fixing this, my first attempts were to use the same strategy as Kokkos managed View labels, but I haven't gotten that working yet. If it's useful we could also add a new slice variant without labels which are identified by their AoSoA index only.

As a different approach, we also have a new option Cabana::ParticleList (that needs more documentation/examples) for managing particle fields with type tags and getting data at the particle (thread) level instead of separate slice fields for all particles

Last, adding @masterleinad here who has led the Kokkos SYCL backend development to see if he has suggestions

[masterleinad]

I don't have much to add. You could try explicitly copying the kernel by swapping all defined(KOKKOS_ARCH_INTEL_GPU) with 0 in https://github.com/kokkos/kokkos/blob/5ad609661e570ba6aa7716a26a91cb67d559f8a2/core/src/SYCL/Kokkos_SYCL_Instance.hpp#L246-L354.

[XzzX]

Dockerfile.txt
The error can be reproduced with this Dockerfile.
It also has some warnings that can be ignored, i.e. MPI mismatch with HDF5 and floating point warnings.