tool for profiling a GPU run at scale?

[BenWibking]

Hi,

I would like to profile a GPU run at scale to see if I am saturating the network bandwidth. I am doing weak scaling tests out to 64 nodes / 256 GPUs on a system that has 4 V100s (32GB model) per node, with 1 (100 Gb/sec) Infiniband card per node.

Nsight Systems doesn't seem like the right tool for this task, and I don't have access to a Cray system. Is there another option that works reasonably well? The links to IPM in the documentation (https://amrex-codes.github.io/amrex/docs_html/External_Profiling_Tools.html#ipm-cross-platform-integrated-performance-monitoring) seem to be dead.

Thanks,
Ben

[n8xm]

FWIW, the link to the IPM SourceForge sourceforge page seems to work. The link to the NERSC page appears to be dead, however, I was able to find an archived copy on Wayback Machine.

I also found the IPM GitHub page, though the last release was in 2013, and the last commit to the master branch was in 2017...

[BenWibking]

Yes, I've managed to run IPM. A lot of functionality seems to be broken, though. I've switched to using HPCToolkit.