Parallel performance #707
Replies: 7 comments
-
|
Beta Was this translation helpful? Give feedback.
-
Thanks for the follow up:
|
Beta Was this translation helpful? Give feedback.
-
What version of CGNS are you using? Would you happen to know what function is taking the time? Is it the actual read call? Is this on a Lustre or GPFS parallel filesystem? If Lustre, what are your stripe size and count? Does your system have the Darshan logs available? |
Beta Was this translation helpful? Give feedback.
-
This is the git development version of CGNS, but I see the same with version 4.3.0. Really the only function I'm using is:
Three times per zone. There are 6588 zones, but they are splits across the 244 processes in this case (so 27 zones per process). This is Lustre, with 1MB stripes and striping across all OSTs, although either way the MPI I/O performance is seconds regardless of striping. We don't have Darshan available by default, but I can run with Darshan if that would be useful. |
Beta Was this translation helpful? Give feedback.
-
Darshan logs would help determine the issue. In the past cgp_open was slow, so it would be good to eliminate the other cgp functions being the issue. |
Beta Was this translation helpful? Give feedback.
-
Running through Darshan and looking at some detailed logs, I can only see
Is this to be expected? |
Beta Was this translation helpful? Give feedback.
-
Turns out, I needed to step I_MPI_EXTRA_FILESYSTEM=1 for the Intel MPI library to get sensible performance. |
Beta Was this translation helpful? Give feedback.
-
Is there any way to optimise performance for larger files. I'm currently reading a ~800MB file into 96 processes using
cgp_
functionality (setupCGP_INDEPENDENT
because there are varying numbers of zones per process) and it's taking minutes to get the data. There are ~2000 zones in the file, but it's not large amounts of data.If I use our MPI-I/O process to read the same data from a binary file rather than a CGNS file I can do it in seconds.
Beta Was this translation helpful? Give feedback.
All reactions