Compiling AMReX with GPU support on Polaris

[rbielaw]

Hey,

I've been trying to compile AMReX on the Polaris system. My understanding was that the site specific makefile should work for the Polaris system. Without GPU acceleration it compiles fine but when trying to compile with the CUDA acceleration I run into an issue with the flags. If anyone has compiled AMReX on Polaris if they can give some guidance on the GNUmakefile settings they used and modules they loaded it would be appreciated.

[zingale]

Here are our notes on building on polaris:
https://amrex-astro.github.io/workflow/alcf.html

[drummerdoc]

And for reasons specific to the combustion work, we needed to explicitly set
CUDA_HOME=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4
and set paths to the math_libs in CUDA, and to lapack and blas (/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/lib)

Hope that helps

[andreasrauch]

Hi,

Thank you for the help so far already, I work with @rbielaw on the same code (that uses AMReX).

I set the CUDA_HOME path in the .bashrc as: export CUDA_HOME=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4

How do I set the paths to math_libs, lapack and blas?

AMReX compiles, but when I try to compile our code that links with AMReX I get the error as shown below:

In file included from /home/ahrauch/amrex/2d_install_dir/include/AMReX_RandomEngine.H:14, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_Random.H:8, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_Utility.H:15, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_RealVect.H:9, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_RealBox.H:10, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_Geometry.H:8, from /home/ahrauch/amrex/2d_install_dir/include/AMReX_AmrLevel.H:7, from /home/ahrauch/um_hmm_chemistry_cantera/include/HMM.H:16, from HMM_thermo.cpp:1: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/math_libs/include/curand.h:59:10: fatal error: cuda_runtime.h: No such file or directory 59 | #include <cuda_runtime.h>

[WeiqunZhang]

Could you provide more details? For example, do make ....whatever_flags_you_use >& make.txt and show what's in make.txt.

[WeiqunZhang]

Are you using amrex's gnu make files when you compile your own code? If so, it's probably easier to do a super build that compiles amrex and your code together, not separately.

[andreasrauch]

make.txt

Output from make is attached. No I'm not using amrex's gnu make files when I compile my code. A super build does sound easier, but is not how we've set up the code right now. The modules loaded are:

`ahrauch@polaris-login-02:~/um_hmm_chemistry_cantera/src> module list

Currently Loaded Modules:

1. craype-x86-rome 4) perftools-base/22.05.0 7) craype/2.7.15 10) cray-pmi/6.1.2 13) cray-libpals/1.1.7
2. libfabric/1.11.0.4.125 5) craype-accel-nvidia80 8) cray-dsmml/0.2.2 11) cray-pmi-lib/6.0.17 14) PrgEnv-gnu/8.3.3
3. craype-network-ofi 6) gcc/11.2.0 9) cray-mpich/8.1.16 12) cray-pals/1.1.7 15) nvhpc-mixed/21.9`

[WeiqunZhang]

Looks like it's using CC, a C++ compiler wrapper, to compile CUDA code. I think you need to use nvcc.