prov/cxi performance regression in fi_pingpong

[philippfriese]

Describe the bug
Using the upstreamed CXI provider (as of commit fc869ae main branch) yields reduced throughput in fi_pingpong (14GB/s for ofiwg/libfabric compared to 20GB/s for HPE-internal libfabric).

To Reproduce
Steps to reproduce the behavior:

• Launch fi_pingpong -p cxi -e rdm on two Slingshot-connected nodes.
• Observe performance deviation between ofiwg/libfabric and HPE-internal libfabric

Expected behavior
Equivalent performance between both libfabric-variants (~20GB/s).

Output
Deviating performance:

• ~14GB/s for ofiwg/libfabric
• ~20GB/s for hpe/libfabric

It is worth noting that the observed throughput of ofiwg/libfabric can be increased by setting the number of iterations from the default 10 to 100 via -I 100.
Additionally, using osu_bw and osu_latency from the OSU Microbenchmark Suite, no performance differences are observed between the two libfabric variants.

I've attached raw output of the fi_pingpong runs and osu_bw/osu_latency runs.

Environment:

• ofiwg/libfabric at commit fc869ae main branch
• ofiwg/libfabric configure setup: ./configure LDFLAGS=-Wl,--build-id --enable-cxi=yes --enable-only --enable-restricted-dl --enable-tcp --enable-udp --enable-rxm --enable-rxd --enable-hook_debug --enable-hook_hmem --enable-dmabuf_peer_mem --enable-verbs --enable-gdrcopy-dlopen --enable-profile=dl
• hpe/libfabric version 1.15
• OpenMPI 4.1.6 with --with-ofi=yes
• OSU Microbenchmark: 7.3
• openSUSE Leap 15.5, kernel 5.14.21
• aarch64

Additional context
Due to a currently unresolved issue with the local Slingshot deployment on the used ARM platform, it is required to set FI_CXI_LLRING_MODE=never for both fi_pingpong and osu_bw.

[SSSSeb]

ping @mindstorm38

[mindstorm38]

I can't reproduce the regression, here's my environment:

• HPE/libfabric v1.20.1
• OFI/libfabric: v1.21.0a1 (your commit)
• Iterations: 100
• Aarch64 (also works with x86_64)
• With FI_CXI_LLRING_MODE=never
• Both are manually built from sources

Not yet tested with MPI

[lflis]

@mindstorm38
Which slingshot libraries version are you using?

[mindstorm38]

I'm using the latest internal sources, I don't know the version number to be honest. I configure cxi, cassini headers and UAPI headers to directly point to the sources. Please tell me if you have any command to check a version that would be interesting for you, but note that my installation is not standard compared to official SlingShot packages, I'm working in parallel on a packages-based installation but it's on x86_64 so this will not be helpful in this case I guess (I'll try anyway).