parallel reading polyhedral meshes

[jaapGit]

Hi all,

I am currently working on reading and writing cgns data in parallel using the cgns lib. Until now we where reading the data on the master proc and distributed it to the other procs but this has its limits with respect to memory usage and performance. Is it, using the latest cgns version, possible to read and write polyhedral meshes in parallel? It seems that some cgns routines are missing:

cgp_poly_elements_read_data_f

routines to read and write the parent data

Or is my understanding incorrect?

Best regards

Jaap Windt

[brtnfld]

I'm working on the missing Fortran APIs here:

https://github.com/brtnfld/CGNS/tree/CGNS-276

[ref] #671

[jaapGit]

Hmm, maybe I did something wrong then. As far as I know I used standard 4 files. I will look into it again and will redo my valgrind massif tests as soon as I have time. Thanks for the info.

[jaapGit]

With respect to Element connectivity the midlevel library mentions: "In most cases, the data is not duplicated in the mid-level library and written directly from the user data to disk.The exception to this is in the case of MIXED, NGON_n, and NFACE_n element sets. Even if the size of the element connectivity array can be known through ConnectOffset, the MLL will keep a copy of the data in memory for the partial writes."

This suggests to confirm my valgrind massif test, right?

[MicK7]

Indeed after checking the code, the offset array is fully stored in memory but the elements are not. So you are right there is still room for improvement.

[MicK7]

My conclusion were for the reading part. For the writing part, you need to use cgp_poly_section_write followed by cgp_poly_elements_write_data ... they should not store the whole data ... but as you mentionned valgrind test is needed to check that.

[jaapGit]

Thanks for the response.

For my goal it is a problem that the offset array is fully stored. For now we read the complete data and scatter it to the processes. The same is done for writing. This is fast but for the future it is not good enough because it uses to much memory. Therefore I want to try reading in parallel. If the offset is fully stored the memory problems stays, although it is a bit smaller (which is not my impression from valgrind).

The cgp_poly_elements_write_data routines do not work with NFACE_n data: "These routines only work for constant sized elements, since it is not possible to compute file offsets for variable sized elements without knowledge of the entire element connectivity data" which is indeed confirmed when I tried them.

By the way, you mentioned that all the cg_ routines can be used in parallel too (compiling the lib with DCGNS_ENABLE_PARALLEL:BOOL=ON and use cgp_open, right?). I did not know that, good to hear, thanks. I thought that one had to use the cgp_ routines for parallel reading of coordinates, elements data, etc. I am not sure but I think it was like that in the past. So the cgp_ routines are not needed expect when you want to use Lustre for example?

Something else: If the number of processes is larger than the number of elements in the data set (this happens with unstructured grids easily for one of the boundaries) the parallel reading fails because the requested size (zero) is invalid. It was also not possible, obviously, to skip the reading on some processes because then the program hangs.

best regards,

Jaap