Vanishing Jacobian

[tonyzahtila]

I am trying to run an example case, no matter which I select, be it turbPipe or ethier, I get a vanishing Jacobian error.

If someone else is able to confirm the example cases run, my guess would be it is a precision error, I was wondering if it's possible that by including CUDA I have inadvertently switched to single-precision.

[tonyzahtila]

I am wondering if this is plausibly an issue with the version of MPI, Cuda that I am using?

The modules that I load on my cluster are:
module load OpenMPI/3.1.0-intel-2017.u2-GCC-6.2.0-cuda9-ucx
module load CMake/3.12.1-intel-2017.u2-GCC-6.2.0-CUDA9

[tonyzahtila]

For anybody curious, I have solved this problem. A vanishing Jacobian error is indeed dependent on a poor choice of Intel/GCC compilers. I am not sure that providing more detail will be helpful as I imagine varies per cluster.

Tony

[stgeke]

Can you please explain what compilers you used for Fortran, C and C++ when it was failing.

[tonyzahtila]

Hi Stgeke, I will give it a shot but I do not have much training in computer science.

I first compile nekrs, here I keep the nekrs file from the github repo.

: ${NEKRS_CC:="mpicc"}
: ${NEKRS_CXX:="mpic++"}
: ${NEKRS_FC:="mpif77"}

Here, I load in mpi/cmake libraries that depend on GCC compilers.

I then change to my actual case, I find that in order for the vanishing Jacobian error I must now load in mpi/cmake libraries that depend on Intel compilers.

This allows me to successfully run nekRS with 1 CPU/GPU.

However, working with the system admin on the cluster that I use, he has been able to get the job running with 2 or 4 CPU/GPUs, however, when he has compiled these, we again get the Vanishing Jacobians error.

Any suggestions? Do you have any advice on an OpenMPI/Cuda package that we can use on our system architecture? We have NVIDIA P100 GPUs and Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz.

[richardnixonshead]

I am the system admin for the cluster Tony is running on.

We have the Vanish Jacobian error when using GCC 8.3.0, CUDA 10.1.243, OpenMPI 3.1.4. We're using GCC as CC, CXX and FC. We compile OpenMPI without CUDA support, but with UCX 1.10.1 from OFED 5.2-1.0.4.0.

Let me know if you need anything else.

[stgeke]

Unfortunately I cannot reproduce this error. Any chance to get access to this machine? In this case please contact me by email.

[richardnixonshead]

Are you able to send me the output of

$ ldd ~/.local/nekrs/lib/libnekrs.so
linux-vdso.so.1 => (0x00007fff506bf000)
libocca.so => /home/tzahtila/.local/nekrs/lib/libocca.so (0x00002b6f63925000)
libmpi_usempif08.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_usempif08.so.40 (0x00002b6f63412000)
libmpi_usempi_ignore_tkr.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_usempi_ignore_tkr.so.40 (0x00002b6f6344a000)
libmpi_mpifh.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_mpifh.so.40 (0x00002b6f63459000)
libgfortran.so.5 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgfortran.so.5 (0x00002b6f63c15000)
libquadmath.so.0 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libquadmath.so.0 (0x00002b6f634cb000)
libmpi_cxx.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_cxx.so.40 (0x00002b6f6350c000)
libmpi.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi.so.40 (0x00002b6f63e84000)
libstdc++.so.6 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libstdc++.so.6 (0x00002b6f63f8b000)
libm.so.6 => /lib64/libm.so.6 (0x00002b6f64125000)
libgomp.so.1 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgomp.so.1 (0x00002b6f63542000)
libgcc_s.so.1 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgcc_s.so.1 (0x00002b6f63572000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b6f64427000)
libc.so.6 => /lib64/libc.so.6 (0x00002b6f64643000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b6f64a11000)
libcuda.so.1 => not found
libopen-rte.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libopen-rte.so.40 (0x00002b6f64c15000)
libopen-pal.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libopen-pal.so.40 (0x00002b6f64ccf000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002b6f64d93000)
librt.so.1 => /lib64/librt.so.1 (0x00002b6f64f96000)
libcudart.so.10.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/cuda/10.1.243/lib64/libcudart.so.10.1 (0x00002b6f6519e000)
libhwloc.so.5 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/hwloc/1.11.12/lib/libhwloc.so.5 (0x00002b6f6358f000)
libnuma.so.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/numactl/2.0.12/lib/libnuma.so.1 (0x00002b6f635d1000)
libpciaccess.so.0 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libpciaccess/0.14/lib/libpciaccess.so.0 (0x00002b6f635de000)
libxml2.so.2 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libxml2/2.9.9/lib/libxml2.so.2 (0x00002b6f6541a000)
liblzma.so.5 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/xz/5.2.4/lib/liblzma.so.5 (0x00002b6f65584000)
libz.so.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/zlib/1.2.11/lib/libz.so.1 (0x00002b6f635e9000)
/lib64/ld-linux-x86-64.so.2 (0x00002b6f633ed000)

$ ldd ~/.local/nekrs/bin/nekrs
linux-vdso.so.1 => (0x00007ffd08be4000)
libnekrs.so => /home/tzahtila/.local/nekrs/lib/libnekrs.so (0x00002af08fdfb000)
libocca.so => /home/tzahtila/.local/nekrs/lib/libocca.so (0x00002af09010f000)
libmpi_cxx.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_cxx.so.40 (0x00002af08fbfc000)
libmpi.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi.so.40 (0x00002af08fc1d000)
libstdc++.so.6 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libstdc++.so.6 (0x00002af0903ff000)
libm.so.6 => /lib64/libm.so.6 (0x00002af090599000)
libgomp.so.1 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgomp.so.1 (0x00002af08fd3a000)
libgcc_s.so.1 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgcc_s.so.1 (0x00002af08fd69000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002af09089b000)
libc.so.6 => /lib64/libc.so.6 (0x00002af090ab7000)
libmpi_usempif08.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_usempif08.so.40 (0x00002af08fd83000)
libmpi_usempi_ignore_tkr.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_usempi_ignore_tkr.so.40 (0x00002af08fdbb000)
libmpi_mpifh.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libmpi_mpifh.so.40 (0x00002af090e85000)
libgfortran.so.5 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libgfortran.so.5 (0x00002af090ef6000)
libquadmath.so.0 => /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64/libquadmath.so.0 (0x00002af091165000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002af0911a6000)
libcuda.so.1 => not found
libopen-rte.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libopen-rte.so.40 (0x00002af0913aa000)
libopen-pal.so.40 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc-cuda/8.3.0-10.1.243/openmpi/3.1.4/lib/libopen-pal.so.40 (0x00002af091464000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002af091528000)
librt.so.1 => /lib64/librt.so.1 (0x00002af09172b000)
libcudart.so.10.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/cuda/10.1.243/lib64/libcudart.so.10.1 (0x00002af091933000)
libhwloc.so.5 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/hwloc/1.11.12/lib/libhwloc.so.5 (0x00002af091baf000)
libnuma.so.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/numactl/2.0.12/lib/libnuma.so.1 (0x00002af08fdce000)
libpciaccess.so.0 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libpciaccess/0.14/lib/libpciaccess.so.0 (0x00002af08fddb000)
libxml2.so.2 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libxml2/2.9.9/lib/libxml2.so.2 (0x00002af091bf1000)
libz.so.1 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/zlib/1.2.11/lib/libz.so.1 (0x00002af091d5b000)
liblzma.so.5 => /usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/xz/5.2.4/lib/liblzma.so.5 (0x00002af091d73000)
/lib64/ld-linux-x86-64.so.2 (0x00002af08fbd7000)

[richardnixonshead]

@stgeke Any way we could get the ldd output from your working install?

[stgeke]

Please contact me by email.

…

On Fri, Mar 19, 2021 at 11:00 AM Sean Crosby ***@***.***> wrote: @stgeke <https://github.com/stgeke> Any way we could get the ldd output from your working install? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#245 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M772JG5BGG23ND4XFSLTEMODBANCNFSM4Y73OKCA> .

[stgeke]

This was related to bug fixed here:
Nek5000/Nek5000@e3fb122