Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression when *linking* to Ascent #1225

Open
pgrete opened this issue Nov 21, 2023 · 2 comments
Open

Performance regression when *linking* to Ascent #1225

pgrete opened this issue Nov 21, 2023 · 2 comments
Labels
bug
Milestone
0.9.4

Comments

@pgrete
Copy link

pgrete commented Nov 21, 2023

On Lumi (same architecture as Frontier) I am observing a significant performance regression (2x slower) when linking to Ascent (not even calling any function) in AthenaPK/Parthenon.

I looked at some profiling data and it seems that compute kernels are not affected and that the performance regression stems stem waiting for communication between ranks to finish (so it's independent of any startup cost, IO, ...).

Has anyone observed sth similar?
Does anything special happen to MPI when Ascent is being linked in or to the runtime configuration?

My software env on Lumi looks like:

$ module list

Currently Loaded Modules:
  1) ModuleLabel/label                     (S)  13) cce/15.0.1                       25) ncurses/6.4-cpeCray-23.03      37) util-linux/2.38.1-cpeCray-23.03
  2) lumi-tools/23.11                      (S)  14) perftools-base/23.03.0           26) LAME/3.100-cpeCray-23.03       38) Brotli/1.0.9-cpeCray-23.03
  3) init-lumi/0.2                         (S)  15) rocm/5.2.3                       27) x264/20230226-cpeCray-23.03    39) freetype/2.12.1-cpeCray-23.03
  4) LUMI/23.03                            (S)  16) buildtools/23.03-bootstrap  (H)  28) x265/3.5-cpeCray-23.03         40) fontconfig/2.14.1-cpeCray-23.03
  5) craype-x86-trento                          17) craype/2.7.20                    29) expat/2.4.9-cpeCray-23.03      41) libpciaccess/0.17-cpeCray-23.03
  6) craype-accel-amd-gfx90a                    18) cray-mpich/8.1.27           (H)  30) libreadline/8.2-cpeCray-23.03  42) xorg-macros/1.19.3
  7) libfabric/1.15.2.0                         19) cray-libsci/23.02.1.1            31) ICU/72.1-cpeCray-23.03         43) PCRE2/10.40-cpeCray-23.03
  8) craype-network-ofi                         20) cray-dsmml/0.2.2                 32) XZ/5.4.2-cpeCray-23.03         44) X11/23.03-cpeCray-23.03
  9) xpmem/2.5.2-2.4_3.50__gd0f7936.shasta      21) cpeCray/23.03                    33) libxml2/2.9.14-cpeCray-23.03   45) FriBidi/1.0.12-cpeCray-23.03
 10) partition/G                           (S)  22) zlib/1.2.12-cpeCray-23.03        34) libxslt/1.1.37-cpeCray-23.03   46) FFmpeg/5.1.2-cpeCray-23.03
 11) cray-hdf5-parallel/1.12.2.3                23) libpng/1.6.38-cpeCray-23.03      35) gettext/0.21.1-cpeCray-23.03
 12) cray-python/3.9.13.1                       24) bzip2/1.0.8-cpeCray-23.03        36) file/5.43-cpeCray-23.03

And here are the libraries being linked for a build without Ascent:

	linux-vdso.so.1 (0x00007f7878f88000)
	libkokkoscontainers.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkoscontainers.so.4.1 (0x00007f7878f6e000)
	libkokkoscore.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkoscore.so.4.1 (0x00007f7878e2b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f7878e0e000)
	libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1 (0x00007f78789c4000)
	libamdhip64.so.5 => /opt/rocm/lib/libamdhip64.so.5 (0x00007f7877a96000)
	libkokkossimd.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkossimd.so.4.1 (0x00007f7877a8f000)
	libhdf5_parallel_crayclang.so.200 => /opt/cray/pe/lib64/libhdf5_parallel_crayclang.so.200 (0x00007f7877642000)
	libmpi_cray.so.12 => /opt/cray/pe/lib64/libmpi_cray.so.12 (0x00007f7874ac9000)
	libmpi_gtl_hsa.so.0 => /opt/cray/pe/lib64/libmpi_gtl_hsa.so.0 (0x00007f7874851000)
	libxpmem.so.0 => /opt/cray/xpmem/default/lib64/libxpmem.so.0 (0x00007f787484e000)
	libstdc++.so.6 => /opt/cray/pe/gcc-libs/libstdc++.so.6 (0x00007f787442c000)
	libquadmath.so.0 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libquadmath.so.0 (0x00007f78741e3000)
	libmodules.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libmodules.so.1 (0x00007f78741ca000)
	libfi.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libfi.so.1 (0x00007f7873c25000)
	libcraymath.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libcraymath.so.1 (0x00007f7873b3e000)
	libf.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libf.so.1 (0x00007f7873aaa000)
	libu.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libu.so.1 (0x00007f78739a1000)
	libcsup.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libcsup.so.1 (0x00007f7873996000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7873972000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f7873826000)
	libunwind.so.1 => /opt/cray/pe/cce/15.0.1/cce-clang/x86_64/lib/libunwind.so.1 (0x00007f7873811000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f787361a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7878f8a000)
	libelf.so.1 => /usr/lib64/libelf.so.1 (0x00007f78733ff000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f78733f5000)
	libdrm.so.2 => /appl/lumi/SW/LUMI-23.03/G/EB/X11/23.03-cpeCray-23.03/lib/libdrm.so.2 (0x00007f78733d8000)
	libdrm_amdgpu.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/X11/23.03-cpeCray-23.03/lib/libdrm_amdgpu.so.1 (0x00007f78733c9000)
	libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f78731bd000)
	libgcc_s.so.1 => /opt/cray/pe/gcc-libs/libgcc_s.so.1 (0x00007f7872f9e000)
	libamd_comgr.so.2 => /opt/rocm/lib/libamd_comgr.so.2 (0x00007f786b8da000)
	libz.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/zlib/1.2.12-cpeCray-23.03/lib/libz.so.1 (0x00007f786b8b5000)
	libfabric.so.1 => /opt/cray/libfabric/1.15.2.0/lib64/libfabric.so.1 (0x00007f786b5c2000)
	libatomic.so.1 => /opt/cray/pe/gcc-libs/libatomic.so.1 (0x00007f786b3b9000)
	libpmi.so.0 => /opt/cray/pe/lib64/libpmi.so.0 (0x00007f786b1b7000)
	libpmi2.so.0 => /opt/cray/pe/lib64/libpmi2.so.0 (0x00007f786af93000)
	libgfortran.so.5 => /opt/cray/pe/gcc-libs/libgfortran.so.5 (0x00007f786aac8000)
	libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f786a89a000)
	libcxi.so.1 => /usr/lib64/libcxi.so.1 (0x00007f786a673000)
	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f786a5c9000)
	libjson-c.so.3 => /usr/lib64/libjson-c.so.3 (0x00007f786a3b9000)
	libpals.so.0 => /opt/cray/pe/lib64/libpals.so.0 (0x00007f786a1b1000)
	libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007f7869f89000)
	libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f7869d6a000)
	libssh.so.4 => /usr/lib64/libssh.so.4 (0x00007f7869afc000)
	libpsl.so.5 => /usr/lib64/libpsl.so.5 (0x00007f78698ea000)
	libssl.so.1.1 => /usr/lib64/libssl.so.1.1 (0x00007f786984b000)
	libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007f786950c000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f78694b9000)
	libldap_r-2.4.so.2 => /usr/lib64/libldap_r-2.4.so.2 (0x00007f7869464000)
	liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007f7869452000)
	libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007f7869321000)
	libbrotlidec.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/Brotli/1.0.9-cpeCray-23.03/lib/libbrotlidec.so.1 (0x00007f786930f000)
	libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007f7869101000)
	libunistring.so.2 => /usr/lib64/libunistring.so.2 (0x00007f7868d7e000)
	libjitterentropy.so.3 => /usr/lib64/libjitterentropy.so.3 (0x00007f7868b75000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f7868a9b000)
	libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f7868a83000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f786887f000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f7868870000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f7868858000)
	libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007f7868639000)
	libbrotlicommon.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/Brotli/1.0.9-cpeCray-23.03/lib/libbrotlicommon.so.1 (0x00007f7868614000)
	libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f786840f000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f78681e6000)
	libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f7867f5d000)

and with Ascent:

	linux-vdso.so.1 (0x00007fff47bdf000)
	libascent_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libascent_mpi.so (0x00007fe71796a000)
	libascent_flow.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libascent_flow.so (0x00007fe71793c000)
	librover_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/librover_mpi.so (0x00007fe717891000)
	libdray_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libdray_mpi.so (0x00007fe7168ed000)
	libRAJA.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/raja-v2022.10.4/lib/libRAJA.so (0x00007fe7168d6000)
	libumpire.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/umpire-2022.10.0/lib/libumpire.so (0x00007fe7167ed000)
	libmfem.so.4.5.2 => /pfs/lustrep1/projappl/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/mfem-4.5.2/lib/libmfem.so.4.5.2 (0x00007fe714dfd000)
	libconduit.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit.so (0x00007fe714bd0000)
	libconduit_relay.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit_relay.so (0x00007fe714ace000)
	libconduit_blueprint.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit_blueprint.so (0x00007fe71462b000)
	libcamp.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/camp-2022.10.1/lib/libcamp.so (0x00007fe714625000)
	libapcomp_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libapcomp_mpi.so (0x00007fe714502000)
	libconduit_relay_mpi_io.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit_relay_mpi_io.so (0x00007fe71446c000)
	libconduit_blueprint_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit_blueprint_mpi.so (0x00007fe7143c1000)
	libconduit_relay_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/conduit-v0.8.8/lib/libconduit_relay_mpi.so (0x00007fe7143a3000)
	libvtkh_rendering_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libvtkh_rendering_mpi.so (0x00007fe7142b3000)
	libvtkh_filters_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libvtkh_filters_mpi.so (0x00007fe713e22000)
	libvtkh_compositing_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libvtkh_compositing_mpi.so (0x00007fe713d53000)
	libvtkh_core_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libvtkh_core_mpi.so (0x00007fe713cfa000)
	libvtkh_utils_mpi.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libvtkh_utils_mpi.so (0x00007fe713cc5000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fe713ca3000)
	libascent_lodepng.so => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/ascent-develop/lib/libascent_lodepng.so (0x00007fe713c47000)
	libvtkm_filter_connected_components-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_connected_components-2.0.so.2.0 (0x00007fe713aca000)
	libvtkm_filter_contour-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_contour-2.0.so.2.0 (0x00007fe71189a000)
	libvtkm_filter_vector_analysis-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_vector_analysis-2.0.so.2.0 (0x00007fe70dd27000)
	libvtkm_filter_mesh_info-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_mesh_info-2.0.so.2.0 (0x00007fe70c98f000)
	libvtkm_filter_density_estimate-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_density_estimate-2.0.so.2.0 (0x00007fe70c5ff000)
	libvtkm_filter_field_conversion-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_field_conversion-2.0.so.2.0 (0x00007fe70c133000)
	libvtkm_filter_field_transform-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_field_transform-2.0.so.2.0 (0x00007fe70bcb8000)
	libvtkm_filter_flow-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_flow-2.0.so.2.0 (0x00007fe70b5d0000)
	libvtkm_filter_geometry_refinement-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_geometry_refinement-2.0.so.2.0 (0x00007fe70aabf000)
	libvtkm_filter_multi_block-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_multi_block-2.0.so.2.0 (0x00007fe70aa66000)
	libvtkm_filter_resampling-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_resampling-2.0.so.2.0 (0x00007fe70a434000)
	libvtkm_filter_scalar_topology-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_scalar_topology-2.0.so.2.0 (0x00007fe7097c9000)
	libvtkm_filter_zfp-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_zfp-2.0.so.2.0 (0x00007fe7096d1000)
	libvtkm_rendering-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_rendering-2.0.so.2.0 (0x00007fe707ed2000)
	libvtkm_filter_entity_extraction-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_entity_extraction-2.0.so.2.0 (0x00007fe707839000)
	libvtkm_filter_clean_grid-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_clean_grid-2.0.so.2.0 (0x00007fe707733000)
	libvtkm_filter_image_processing-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_image_processing-2.0.so.2.0 (0x00007fe7073e6000)
	libvtkm_filter_core-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_filter_core-2.0.so.2.0 (0x00007fe707306000)
	libvtkm_worklet-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_worklet-2.0.so.2.0 (0x00007fe7070a3000)
	libvtkm_io-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_io-2.0.so.2.0 (0x00007fe706f5c000)
	libvtkm_cont-2.0.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkm_cont-2.0.so.2.0 (0x00007fe70575f000)
	libvtkmdiympi.so.2.0 => /project/project_465000552/env/src/ascent/scripts/build_ascent/CC-Lumi23.03/install/vtk-m-v2.0.0/lib/libvtkmdiympi.so.2.0 (0x00007fe705753000)
	libkokkoscontainers.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkoscontainers.so.4.1 (0x00007fe705741000)
	libkokkoscore.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkoscore.so.4.1 (0x00007fe7055fe000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe7055f9000)
	libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1 (0x00007fe7051ad000)
	libamdhip64.so.5 => /opt/rocm/lib/libamdhip64.so.5 (0x00007fe70427f000)
	libkokkossimd.so.4.1 => /project/project_465000552/env/src/kokkos/install-4.1.00-CC-LUMI23.03/lib64/libkokkossimd.so.4.1 (0x00007fe70427a000)
	libhdf5_parallel_crayclang.so.200 => /opt/cray/pe/lib64/libhdf5_parallel_crayclang.so.200 (0x00007fe703e2d000)
	libmpi_cray.so.12 => /opt/cray/pe/lib64/libmpi_cray.so.12 (0x00007fe7012b4000)
	libmpi_gtl_hsa.so.0 => /opt/cray/pe/lib64/libmpi_gtl_hsa.so.0 (0x00007fe70103c000)
	libxpmem.so.0 => /opt/cray/xpmem/default/lib64/libxpmem.so.0 (0x00007fe701037000)
	libstdc++.so.6 => /opt/cray/pe/gcc-libs/libstdc++.so.6 (0x00007fe700c15000)
	libquadmath.so.0 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libquadmath.so.0 (0x00007fe7009ce000)
	libmodules.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libmodules.so.1 (0x00007fe7009b5000)
	libfi.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libfi.so.1 (0x00007fe700410000)
	libcraymath.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libcraymath.so.1 (0x00007fe700329000)
	libf.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libf.so.1 (0x00007fe700293000)
	libu.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libu.so.1 (0x00007fe70018a000)
	libcsup.so.1 => /opt/cray/pe/cce/15.0.1/cce/x86_64/lib/libcsup.so.1 (0x00007fe700181000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe70015d000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fe700011000)
	libunwind.so.1 => /opt/cray/pe/cce/15.0.1/cce-clang/x86_64/lib/libunwind.so.1 (0x00007fe6ffffc000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe6ffe03000)
	libhdf5_hl_parallel_crayclang.so.200 => /opt/cray/pe/lib64/libhdf5_hl_parallel_crayclang.so.200 (0x00007fe6ffdda000)
	libdsmml.so.0 => /opt/cray/pe/lib64/libdsmml.so.0 (0x00007fe6ffbcf000)
	libsci_cray_mpi.so.5 => /opt/cray/pe/lib64/libsci_cray_mpi.so.5 (0x00007fe6ff186000)
	libsci_cray.so.5 => /opt/cray/pe/lib64/libsci_cray.so.5 (0x00007fe6fb23e000)
	libpgas-shmem.so.2 => /opt/cray/pe/lib64/cce/libpgas-shmem.so.2 (0x00007fe6fb138000)
	libatomic.so.1 => /opt/cray/pe/gcc-libs/libatomic.so.1 (0x00007fe6faf2f000)
	libgcc_s.so.1 => /opt/cray/pe/gcc-libs/libgcc_s.so.1 (0x00007fe6fad10000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe71829d000)
	libelf.so.1 => /usr/lib64/libelf.so.1 (0x00007fe6faaf1000)
	libdrm.so.2 => /appl/lumi/SW/LUMI-23.03/G/EB/X11/23.03-cpeCray-23.03/lib/libdrm.so.2 (0x00007fe6faad4000)
	libdrm_amdgpu.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/X11/23.03-cpeCray-23.03/lib/libdrm_amdgpu.so.1 (0x00007fe6faac5000)
	libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007fe6fa8b9000)
	libamd_comgr.so.2 => /opt/rocm/lib/libamd_comgr.so.2 (0x00007fe6f31f5000)
	libz.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/zlib/1.2.12-cpeCray-23.03/lib/libz.so.1 (0x00007fe6f31d0000)
	libfabric.so.1 => /opt/cray/libfabric/1.15.2.0/lib64/libfabric.so.1 (0x00007fe6f2edd000)
	libpmi.so.0 => /opt/cray/pe/lib64/libpmi.so.0 (0x00007fe6f2cdb000)
	libpmi2.so.0 => /opt/cray/pe/lib64/libpmi2.so.0 (0x00007fe6f2ab9000)
	libgfortran.so.5 => /opt/cray/pe/gcc-libs/libgfortran.so.5 (0x00007fe6f25ec000)
	libsma.so.0 => /opt/cray/pe/lib64/libsma.so.0 (0x00007fe6f1dae000)
	libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fe6f1b7e000)
	libcxi.so.1 => /usr/lib64/libcxi.so.1 (0x00007fe6f1959000)
	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007fe6f18af000)
	libjson-c.so.3 => /usr/lib64/libjson-c.so.3 (0x00007fe6f169f000)
	libpals.so.0 => /opt/cray/pe/lib64/libpals.so.0 (0x00007fe6f1497000)
	libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007fe6f126d000)
	libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007fe6f1050000)
	libssh.so.4 => /usr/lib64/libssh.so.4 (0x00007fe6f0de2000)
	libpsl.so.5 => /usr/lib64/libpsl.so.5 (0x00007fe6f0bd0000)
	libssl.so.1.1 => /usr/lib64/libssl.so.1.1 (0x00007fe6f0b31000)
	libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007fe6f07f2000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007fe6f079d000)
	libldap_r-2.4.so.2 => /usr/lib64/libldap_r-2.4.so.2 (0x00007fe6f0748000)
	liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007fe6f0738000)
	libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007fe6f0607000)
	libbrotlidec.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/Brotli/1.0.9-cpeCray-23.03/lib/libbrotlidec.so.1 (0x00007fe6f05f5000)
	libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fe6f03e7000)
	libunistring.so.2 => /usr/lib64/libunistring.so.2 (0x00007fe6f0062000)
	libjitterentropy.so.3 => /usr/lib64/libjitterentropy.so.3 (0x00007fe6efe5b000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007fe6efd81000)
	libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fe6efd69000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe6efb65000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007fe6efb56000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe6efb3c000)
	libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007fe6ef91f000)
	libbrotlicommon.so.1 => /appl/lumi/SW/LUMI-23.03/G/EB/Brotli/1.0.9-cpeCray-23.03/lib/libbrotlicommon.so.1 (0x00007fe6ef8fa000)
	libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007fe6ef6f5000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe6ef4cc000)
	libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fe6ef241000)
@nicolemarsaglia
Copy link
Contributor

Wow, that is a very unfortunate bug. I'm sorry you're experiencing that.

The only thing I've experienced that comes close is a case of linking Ascent with an AMR code that resulted in a detrimental memory leak. It turns out that if they tailored Ascent's build more to the AMR build specification, then they got things working correctly, but I don't have many details.

@cyrush can chime in next week, when he's back, if he recalls anything like this happening before.

@cyrush
Copy link
Member

cyrush commented Nov 27, 2023

@pgrete, yikes.

Ascent should not be changing MPI performance when not used.

We have not had similar reports, but usually the cuplrit in mysteries like this is a subtle build or link issue.

Here is what is different that isn't directly Ascent related:

 "libhdf5_hl_parallel_crayclang.so.200": "/opt/cray/pe/lib64/libhdf5_hl_parallel_crayclang.so.200",
  "libdsmml.so.0": "/opt/cray/pe/lib64/libdsmml.so.0",
  "libsci_cray_mpi.so.5": "/opt/cray/pe/lib64/libsci_cray_mpi.so.5",
  "libsci_cray.so.5": "/opt/cray/pe/lib64/libsci_cray.so.5",
  "libpgas-shmem.so.2": "/opt/cray/pe/lib64/cce/libpgas-shmem.so.2",
  "libsma.so.0": "/opt/cray/pe/lib64/libsma.so.0"

In other projects - I have hit surprises related to linking libs that override signal handlers.

VTK-m uses logguru, which may install some signal handlers but I think those are explicitly installed during runtime - not a a result of linking.

Could this be a difference in synchronization behavior with MPI?

@cyrush cyrush added the bug label Dec 5, 2023
@cyrush cyrush added this to the 0.9.4 milestone May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
0.9.4
Development

No branches or pull requests
3 participants
@cyrush @pgrete @nicolemarsaglia