nekrs sample test fails with CUDA_ERROR_INVALID_DEVICE #438

nandini1610 · 2022-06-17T01:36:42Z

nandini1610
Jun 17, 2022

Running the sample test mpirun -np 2 nekrs --setup turbPipe.par resulted in the following error running on a GPU server.
Initializing device
terminate called after throwing an instance of 'occa::exception'
what():
---[ Error ]--------------------------------------------------------------------
File : /program/nekrs-22.0/nekRS-22.0/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
Line : 30
Function : device
Message : Device: Creating Device
CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
Stack
12 occa::error(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
10 occa::cuda::device::device(occa::json const&)
9 occa::cuda::cudaMode::newDevice(occa::json const&)
8 occa::device::setup(occa::json const&)
7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
6 device_t::device_t(setupAide&, comm_t&)
5 platform_t::platform_t(setupAide&, int, int)
4 nekrs::setup(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int)
3 nekrs()
2 /lib64/libc.so.6(__libc_start_main+0xf5)
1 nekrs()

yslan · 2022-06-17T02:02:48Z

yslan
Jun 17, 2022

Does you machine has two GPU?
What does nvidia-smi shows?

3 replies

nandini1610 Jun 17, 2022
Author

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

nandini1610 Jun 17, 2022
Author

The last run was done on a node with 1GPU. This time I ran on a node with 2 GPU and the cuda error is gone but still i have this error.
mpirun -np 2 nekrs --setup turbPipe.par
__ ____ _____
____ ___ / /__ / __ / /
/ __ \ / _ \ / //// // /_
/ / / // // ,< / , // /
// // __///||// ||/____/ v22.0.0 (no sha)

MPI tasks: 2

reading par file ...

using NEKRS_HOME: /program/nekrs-22.0/nekrs
using NEKRS_CACHE_DIR: /enc/udev_ijnLT/work/shared/turbPipePeriodic/.cache
using OCCA_CACHE_DIR: /enc/udev_ijnLT/work/shared/turbPipePeriodic/.cache/occa/

Initializing device
active occa mode: CUDA

terminate called after throwing an instance of 'occa::exception'
what():
---[ Error ]--------------------------------------------------------------------
File : /program/nekrs-22.0/nekRS-22.0/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
Line : 351
Function : compileKernel
Message : Error compiling [myDummyKernelName], Command: [nvcc -O3 --fmad=true -lineinfo -arch=sm_70 -fatbin -Xptxas -v -I/program/nekrs-22.0/nekrs/include -I/program/nekrs-22.0/nekrs/include -L/program/nekrs-22.0/nekrs/lib -locca -x cu /enc/udev_ijnLT/work/shared/turbPipePeriodic/.cache/occa/cache/8883875db6215ce2/source.cpp -o /enc/udev_ijnLT/work/shared/turbPipePeriodic/.cache/occa/cache/8883875db6215ce2/binary > /dev/null 2>&1]
Stack
12 occa::error(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)
11 occa::cuda::device::compileKernel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, occa::json const&, occa::io::lock_t&)
10 occa::cuda::device::buildKernelFromProcessedSource(occa::hash_t, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, occa::lang::sourceMetadata_t&, occa::lang::sourceMetadata_t&, occa::json const&, occa::io::lock_t)
9 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, occa::hash_t, bool, occa::json const&)
8 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, occa::hash_t, occa::json const&)
7 occa::device::buildKernel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, occa::json const&) const
6 occa::device::buildKernelFromString(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, occa::json const&) const
5 platform_t::platform_t(setupAide&, int, int)
4 nekrs::setup(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int)
3 nekrs()
2 /lib64/libc.so.6(__libc_start_main+0xf5)
1 nekrs()

nandini1610 Jun 17, 2022
Author

I also upgraded the cuda version and tested and i get this error.
mpirun -np 2 nekrs --setup turbPipe.par

MPI tasks: 2

reading par file ...

using NEKRS_HOME: /program/nekrs-22.0/nekrs
using NEKRS_CACHE_DIR: /enc/udev_DunLT/work/shared/turbPipePeriodic/.cache
using OCCA_CACHE_DIR: /enc/udev_DunLT/work/shared/turbPipePeriodic/.cache/occa/

Initializing device
[[CUDACUDA] mode is not enabled, defaulting to [Serial] mode
] mode is not enabled, defaulting to [Serial] mode
active occa mode: Serial

active occa mode does not match selected backend!
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
Abort(1) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1

yslan · 2022-06-17T22:31:40Z

yslan
Jun 17, 2022

I assume you can run single GPU without issue?

From your nvidia-smi output, I can only find 1 GPU.
First, I'd make sure you do have 2 visible GPU on your "2 gpu-node".
nvidia-smi should show multiple GPU id and their status
The error Error compiling [myDummyKernelName], can be related to many things.
I'd recommend to remove the local cache before next actions whenever you found an error.
```
  (in your case, it's at)
  /enc/udev_DunLT/work/shared/turbPipePeriodic/.cache
```
As for the final error
```
[[CUDACUDA] mode is not enabled, defaulting to [Serial] mode
```
It looks like the OCCA cannot run with cuda successfully.
My guess is, you didn't rebuild the course code after upgrading the cuda version?
```
goto <NEKRS_HOME> 
rm -r build
./nrsconfig
...
```

8 replies

yslan Jun 20, 2022

Same thing here. After changing the compiler, the installed nekrs source including occa will no longer usable.
As long as the occa can't work properly, you will always fall back to SERIAL mode and see the error "active occa mode does not match selected backend!"

The rebuild is needed after each time the compiler changes. (cd <nekrs source folder>; rm -r build; ./nrsconfig; ....; cd <case folder>;rm -r .cache;).
And, I will expect you still get the same error (Error compiling [myDummyKernelName]) when you try to run the example with single GPU
Now, it will show up the command it stucks.
You just need to copy and execute the command causing error without redirect the stderr to background > /dev/null 2>&1.
So, you can get the true error why the compilation fails.

nandini1610 Jun 20, 2022
Author

Everytime I do the instal, it is a fresh installation and it is not reusing any existing one.If you look at the nekrs binaries it shows it has been compiled properly. So I tried recompiling multiple times with different cuda version on gcc 8.3.0, I get the same error
"active occa mode does not match selected backend!"

nandini1610 Jun 20, 2022
Author

yslan Jun 20, 2022

This is weird.
Your nvidia-smi was showing 0 Tesla V100-PCIE.
However, occa info only picks up the OpenCL backend and it shows Tesla K80

nandini1610 Jun 20, 2022
Author

occa info
========+======================+===========================================
CPU(s) | Processor Name | Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
| Memory | 59.96 GB
| Clock Frequency |
| SIMD Instruction Set | AVX2
| SIMD Width | 256 bits
| L1d Cache Size | 32 KB
| L1i Cache Size | 32 KB
| L2 Cache Size | 256 KB
| L3 Cache Size | 45 MB
========+======================+===========================================
OpenCL | Device Name | Tesla K80
| Driver Vendor | NVIDIA
| Platform ID | 0
| Device ID | 0
| Memory | 11.17 GB

nandini1610 · 2022-06-20T22:22:35Z

nandini1610
Jun 20, 2022
Author

Finally after active occa mode : CUDA I got the same nvcc error and as you mentioned was able to track down to missing include dir. Let me know what I can do after this step. Should I recompile?

nvcc -O3 --fmad=true -lineinfo -arch=sm_52 -fatbin -Xptxas -v -I/program/nekrs-22.0/nekrs/include -I/program/nekrs-22.0/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include -I/program/nekrs-22.0/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include-fixed -L/program/nekrs-22.0/nekrs/lib -locca -x cu /enc/udev_DHqBeb/work/shared/examples/turbPipe/.cache/occa/cache/dc511cf8ce7411de/source.cpp -o /enc/udev_DHqBeb/work/shared/examples/turbPipe/.cache/occa/cache/dc511cf8ce7411de/binary
ptxas info : 0 bytes gmem
ptxas info : Compiling entry function '_occa_myDummyKernelName_0' for 'sm_52'
ptxas info : Function properties for _occa_myDummyKernelName_0
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 2 registers, 324 bytes cmem[0]

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nekrs sample test fails with CUDA_ERROR_INVALID_DEVICE #438

{{title}}

Replies: 3 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

nekrs sample test fails with CUDA_ERROR_INVALID_DEVICE #438

nandini1610 Jun 17, 2022

Replies: 3 comments · 11 replies

yslan Jun 17, 2022

nandini1610 Jun 17, 2022 Author

nandini1610 Jun 17, 2022 Author

nandini1610 Jun 17, 2022 Author

yslan Jun 17, 2022

yslan Jun 20, 2022

nandini1610 Jun 20, 2022 Author

nandini1610 Jun 20, 2022 Author

yslan Jun 20, 2022

nandini1610 Jun 20, 2022 Author

nandini1610 Jun 20, 2022 Author

nandini1610
Jun 17, 2022

Replies: 3 comments 11 replies

yslan
Jun 17, 2022

nandini1610 Jun 17, 2022
Author

nandini1610 Jun 17, 2022
Author

nandini1610 Jun 17, 2022
Author

yslan
Jun 17, 2022

nandini1610 Jun 20, 2022
Author

nandini1610 Jun 20, 2022
Author

nandini1610 Jun 20, 2022
Author

nandini1610
Jun 20, 2022
Author