Option to print process mapping and communication provider?

[gsamonds]

I'm searching for an MPICH option similar to what I_MPI_DEBUG=5 in Intel MPI does: prints process process mapping and shows which communication provider (e.g. TCP, infiband) is being used.

Ideally, this would be in the form of an environment variable but I'd be happy to know about any command line parameters too.

Does any option like this exist for MPICH 4? Many thanks.

[hzhou]

You can set MPIR_CVAR_DEBUG_SUMMARY=1 to let process 0 dump some information during init. There isn't a place to dump the process mapping currently, mostly because no one has requested yet :)

[gsamonds]

Hello again. I continued experimenting with the OFI+UCX build today and wanted to ask if there is any mechanism that controls which libraries are required at runtime based on the provider being used.

My current build of MPICH requires all of the shared libraries related to both OFI and UCX regardless of the fabric context. I tried to workaround this by copying these libraries into my MPICH installation, and while this allows jobs to run on machines that don't have these libraries installed, this then causes conflicts on machines that do have their own installation of these libraries.

To be clear, these are the specific libraries I'm referring to:
libucp.so.0
libuct.so.0
libucs.so.0
libucm.so.
libfabric.so.1
libpsm_infinipath.so.1
libpsm2.so.2
librdmacm.so.1
libibverbs.so.1

Is there any way to workaround this, so that the same distribution of MPICH could be used on a single node with no high speed interface, yet also run on an Infiniband cluster? Thanks again for all of the support so far.

[hzhou]

We'll probably need to do our own loading of dynamic libraries and implement something similar to plugin or module systems. That is not available today.

[gsamonds]

So far my tests of MPICH with libfabric have shown comparable performance (within about 1%) of Intel MPI 2019.11 on both Omni-Path (libfabric;psm2) and Infiniband (libfabric;verbs;ofi_rxm) machines.

Getting MPICH to run with UCX has been a bit tricky however. When I run "ucx_info" on one of the compute nodes, I can see many different available transports (rc_verbs, rc_mlx5, dc_mlx5, etc) and the mlx5_0:1 device. However when running MPICH with UCX I get errors:
"UCX ERROR ibv_exp_query_device(mlx5_0) returned 38: Protocol not supported"
and warnings saying that whatever transport I have selected with UCX_TLS is not available, and to use, "one or more of: cma, knem, mm, posix, self, shm, sm, sysv, tcp"

Does this indicate an issue with the version of UCX MPICH was configured with? I configured using --with-ucx=embedded, and MPICH's configure script produced the following UCX config (missing IB modules?):
configure: =========================================================
configure: UCX build configuration:
configure: Build prefix: /nisprod/products1/freeware/central_build_solver/mpich/4.0.2/Linux_2.17/x86_64
configure: Configuration dir: ${prefix}/etc/ucx
configure: Preprocessor flags: -DCPU_FLAGS="" -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src
configure: C compiler: gcc -std=gnu99 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch -Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length -Wnested-externs -Wshadow -Werror=declaration-after-statement
configure: C++ compiler: g++ -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch
configure: Multi-thread: disabled
configure: NUMA support: enabled
configure: MPI tests: disabled
configure: VFS support: yes
configure: Devel headers: no
configure: io_demo CUDA support: no
configure: Bindings: < >
configure: UCS modules: < fuse >
configure: UCT modules: < ib rdmacm cma knem >
configure: CUDA modules: < >
configure: ROCM modules: < >
configure: IB modules: < >
configure: UCM modules: < >
configure: Perf modules: < >
configure: =========================================================

Thanks for any input.

[hzhou]

UCX often needs to be configured on the compute node to be properly configured. Either that or try to build MPICH with external ucx so it can pick up a correctly built ucx library on the compute node.

[gsamonds]

Thanks, building MPICH on one of the compute nodes allowed me to run with UCX and an mlx5 device without errors or warnings.

[raffenet]

prints process mapping

Do you mean the CPU binding? E.g.

[0] MPI startup(): 0       28048    beboplogin2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17}
[0] MPI startup(): 1       28049    beboplogin2  {18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35}

If so, you can try setting HYDRA_TOPO_DEBUG=1.

[gsamonds]

Thanks. Yes, I mean CPU binding/pinning.

That variable doesn't seem to be working for me:
[root@MD-3 mpi_hello]# export HYDRA_TOPO_DEBUG=1
[root@MD-3 mpi_hello]# mpirun -np 2 ./a.out
Hello world from processor MD-3, rank 0 out of 2 processors
Hello world from processor MD-3, rank 1 out of 2 processors

Is there some element of the following configuration that could be leading to that variable not printing anything, or is it only active in a multi-node job?
# mpirun -info
HYDRA build details:
Version: 4.0.2
Release Date: Thu Apr 7 12:34:45 CDT 2022
CC: gcc -std=gnu99
Configure options: '--disable-option-checking' '--prefix=/nisprod/products1/freeware/central_build_solver/mpich/4.0.2/Linux_2.17/x86_64' '--with-libfabric' '--with-ucx' '--with-slurm' '--enable-shared' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc -std=gnu99' 'CFLAGS= ' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= '
Process Manager: pmi
Launchers available: ssh rsh fork slurm ll lsf sge manual persist
Topology libraries available: hwloc
Resource management kernels available: user slurm ll lsf sge pbs cobalt
Demux engines available: poll select

[hzhou]

It only prints the binding info if you launch the process with binding options (-bind-to). By default, hydra doesn't do any binding, thus there is nothing to print.

[gsamonds]

Ah OK, thank you. I'm coming from using Intel MPI most of the time which has pinning enabled by default these days, need to break some of those habits apparently.