Replies: 4 comments 4 replies
-
|
As an example of writing polyhedra in parallel, I came up with the code below a while ago. It has some assumptions, e.g. no shared vertices or faces between ranks, which is unlikely in decomposed CFD grids. You would have to do more bookkeeping to get that correct I think. Maybe the example is useful to get started with parallel cgns? For starters, you don't need to first open the file serially, write the bases and zones, then re-open it in parallel mode with Maybe it can be added to the examples if it is indeed useful. /*
Parallel writing of polyhedral cells to CGNS file.
This requires parallel CGNS and HDF5 libraries.
HDF5 v1.12.1 (latest).
configure with:
./configure --prefix=<path/to/install> --enable-fortran --enable-fortran2003 --enable-hl --enable-parallel --enable-shared
make && make install
CGNS 4.2.0 (latest).
configure with:
mkdir build; cd build
export HDF5_ROOT=<path/to/install>
cmake </path/to/CGNS/source> -DCMAKE_INSTALL_PREFIX=<path/to/install> \
-DCGNS_ENABLE_PARALLEL=1 \
-DCGNS_BUILD_CGNSTOOLS=1 \
-DCGNS_ENABLE_FORTRAN=1 \
-DCGNS_ENABLE_HDF5=1 \
-DHDF5_NEED_MPI=1 \
-DCGNS_BUILD_SHARED=1 \
-DCGNS_USE_SHARED=1 \
-DCGNS_BUILD_CGNSTOOLS=1 \
-DCMAKE_Fortran_FLAGS=-fPIC
make && make install
*/
#include <assert.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#ifdef _WIN32
#include <io.h>
#define unlink _unlink
#else
#include <unistd.h>
#endif
#include "mpi.h"
#include "pcgnslib.h"
void callCGNS(int error) {
if (error != CG_OK)
cgp_error_exit();
}
int main(int argc, char **argv) {
// set up MPI
int comm_size;
int comm_rank;
int ierr = 0;
// parallel/serial read communicators
MPI_Comm comm_parallel = MPI_COMM_WORLD;
MPI_Comm comm_serial = MPI_COMM_SELF;
MPI_Init(NULL, NULL);
MPI_Comm_size(comm_parallel, &comm_size);
MPI_Comm_rank(comm_parallel, &comm_rank);
printf("hello from %d/%d\n", comm_rank, comm_size);
// Every process has 10 vertices to build three polyhedra
// 1) a central cube, and 2/3) two pyramids sticking out
// above and below the cube
#define NNODES 10
double xs[NNODES] = {0, 1, 1, 0, 0, 1, 1, 0, 0.5, 0.5};
double ys[NNODES] = {0, 0, 1, 1, 0, 0, 1, 1, 0.5, 0.5};
double zs[NNODES] = {0, 0, 0, 0, 1, 1, 1, 1, 2.0, -1.0};
#define FACEARRAY_SIZE 48
cgsize_t faces[FACEARRAY_SIZE] = {
1, 2, 3, 4, // cube bottom
5, 6, 7, 8, // cube top
1, 2, 6, 5, // cube front
2, 3, 7, 6, // cube right
3, 4, 8, 7, // cube back
1, 4, 8, 5, // cube left
5, 6, 9, // top pyramid
6, 7, 9, // top pyramid
7, 8, 9, // top pyramid
8, 5, 9, // top pyramid
1, 2, 10, // bottom pyramid
2, 3, 10, // bottom pyramid
3, 4, 10, // bottom pyramid
4, 1, 10 // bottom pyramid
};
#define NFACES 14
cgsize_t face_offsets[1+NFACES] = {
0, // initial offset is zero
4, 8, 12, 16, 20, 24, // cube
27, 30, 33, 36, // top pyramid
39, 42, 45, 48 // bottom pyramid
};
#define CELLARRAY_SIZE 16
cgsize_t cells[CELLARRAY_SIZE] = {
1, 2, 3, 4, 5, 6, //cube
2, 7, 8, 9, 10, // top pyramid
1, 11, 12, 13, 14 // bottom pyramid
};
#define NCELLS 3
cgsize_t cell_offsets[1+NCELLS] = {
0,
6, 11, 16
};
// move each grid away from the others by shifting 4 units in Z-direction
for(int i = 0; i < NNODES; ++i){
zs[i] = zs[i] + 4*comm_rank;
}
// there are no shared faces between grids on different ranks
for(int i = 0; i < FACEARRAY_SIZE; ++i) {
faces[i] = faces[i] + comm_rank * NNODES;
}
for(int i = 0; i < CELLARRAY_SIZE; ++i) {
cells[i] = cells[i] + comm_rank * NFACES;
}
for(int i = 0; i <= NFACES; ++i)
{
face_offsets[i] = face_offsets[i] + comm_rank*FACEARRAY_SIZE;
}
for(int i = 0; i <= NCELLS; ++i)
{
cell_offsets[i] = cell_offsets[i] + comm_rank*CELLARRAY_SIZE;
}
int F, B, Z, Cx, Cy, Cz, S, E;
// open the file
callCGNS(cg_configure(CG_CONFIG_COMPRESS, 0));
callCGNS(cgp_mpi_comm(comm_parallel));
callCGNS(cgp_open("./par-polyhedra.cgns", CG_MODE_WRITE, &F));
// write the base
int cell_dim = 3;
int phys_dim = 3;
callCGNS(cg_base_write(F, "Base_Volume_Elements", cell_dim, phys_dim, &B));
// write the zone
cgsize_t zoneSize[9];
zoneSize[0] = NNODES * comm_size;
zoneSize[1] = NCELLS * comm_size;
zoneSize[2] = 0;
callCGNS(cg_zone_write(F, B, "Zone_Interior", zoneSize, Unstructured, &Z));
// start writing coordinates
callCGNS(cgp_coord_write(F, B, Z, RealDouble, "CoordinateX", &Cx));
callCGNS(cgp_coord_write(F, B, Z, RealDouble, "CoordinateY", &Cy));
callCGNS(cgp_coord_write(F, B, Z, RealDouble, "CoordinateZ", &Cz));
// write coordinate data
cgsize_t start = 1;
cgsize_t end = NNODES;
start += comm_rank*NNODES;
end += comm_rank*NNODES;
callCGNS(cgp_coord_write_data(F, B, Z, Cx, &start, &end, xs));
callCGNS(cgp_coord_write_data(F, B, Z, Cy, &start, &end, ys));
callCGNS(cgp_coord_write_data(F, B, Z, Cz, &start, &end, zs));
// start writing the NGON_n faces section
start = 1;
end = comm_size*NFACES;
cgsize_t offsetsTotalSize = comm_size * FACEARRAY_SIZE;
callCGNS(cgp_poly_section_write(F, B, Z, "Elements 2D", NGON_n, start, end,
offsetsTotalSize, 0, &E));
// write the faces data of this process
start = 1 + comm_rank *NFACES;
end = (1+ comm_rank)*NFACES;
callCGNS(cgp_poly_elements_write_data(F, B, Z, E, start, end, faces, face_offsets));
// start writing the NFACE_n cells section
start = 1;
end = comm_size*NCELLS;
offsetsTotalSize = comm_size * CELLARRAY_SIZE;
callCGNS(cgp_poly_section_write(F, B, Z, "Elements 3D", NFACE_n, start, end,
offsetsTotalSize, 0, &E));
// write the cells data of this process
start = 1 + comm_rank *NCELLS;
end = (1+ comm_rank)*NCELLS;
callCGNS(cgp_poly_elements_write_data(F, B, Z, E, start, end, cells, cell_offsets));
// close the file and finalize MPI
callCGNS(cgp_close(F));
MPI_Finalize();
return 0;
} |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the reply and for the sample script. Does your code work for single-zone files only? When I run the following: The output CGNS file has a single Zone even though all MPI processes call Since, apparently, there is no minimum reproducible example for writing a MULTIZONE .cgns file in parallel, I need to know whether the parallel write is possible only to a single zone. Or, whether it is possible to create a MULTIZONE .cgns file sequentially (meaning, Bases, Zones, Sections, Coordinates, Boundary Conditions, etc but not the DataArray_t nodes), and then in parallel have all ranks make their respective edits. |
Beta Was this translation helpful? Give feedback.
-
|
I don't have a standalone snippet of code to show, but the IOSS library had CGNS input and output in both parallel and serial. It may provide some insights on parallel CGNS output. See https://github.com/sandialabs/seacas/tree/master/packages/seacas/libraries/ioss/src/cgns. The Iocgns_ParallelDatabase.C and Iocgns_DecompositionData.C have most of the parallel read/write calls. It won't be too easy to understand, but it does work and seems to scale fairly well. Apologies for the lack of documentation. |
Beta Was this translation helpful? Give feedback.
-
I missed out on this in the first read, sorry. You could let each process write their own CGNS file, and create a single "whole grid" CGNS file that contains file links to each part. It does mean that you will have a lot of files, but you don't need to use any parallel CNGS routines, and it will scale as well as your storage can process the data. |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Sorry for not replying. If CGNS doesn't offer this capability, then perhaps the merging could be done via HDF5? |
Beta Was this translation helpful? Give feedback.
-
|
I do parallel CGNS with a single file and each process writes a separate zone. Works fine for me. However, the flow to do it is somewhat counter intuitive. I'm using Fortran, the same should work fine for C with the equivalent calls: etc.... The above code, run by all processes, sets up the file structure/metadata. The counter intuitive part is all processes write out all the metadata for all zones. Strange, but seems to be required. (happy to be told otherwise, but that was what was required to get it working for me). Then I close the file (I don't think this is strictly necessary, but at one point I thought it the safest thing to do), reopen it and then write the data. The data can then be written in parallel, and separately, with code like this: etc... Where each process is writing its own zones independently by specifying It took me a while to figure out the specific control flow needed to keep CGNS happy, but the above works for me now. Happy to provide more details if it helps. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the input. I was actually doing the exact same thing (in principle) to get mine started. Instead of having all my processors write-out the meta data, I just have Process-0 do it by itself. Then, I reopen the file, but, not as a CGNS file, rather, as an HDF5 file. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe to write connectivities, you could try cgp_section_write or cgp_poly_section_write to directly create the node in parallel mode. Sizing everything will still be needed but it would prevent a close/open cycle. |
Beta Was this translation helpful? Give feedback.
-
Greetings.
I am trying to develop a CFD code with some colleagues and we looked to CGNS for a file exchange format.
The documentation for sequential CGNS and the few examples available online were enough to get us started for serial I/O.
However, when it comes to the parallel version of the library (PCGNS), the examples are remarkably scanty. The official website provides two versions (one Fortran, the other C) of essentially the same structured, single zone write. The next best thing I found online was a set of slides by a fellow named Hauser, but following his instructions produces buggy code (segmentation faults mostly).
My current objective is to write a mesh that is distributed over multiple MPI processes in parallel to .cgns format across many Zones. I understand that composing the file structure (i.e., the nodes) is not inherently a parallel/concurrent activity. So that I will relegate to a single process. However, once the structure is set, I should be able to concurrently tell a bunch of MPI processes to write their data (coordinates, flow solution) simultaneously. This I cannot seem to do.
The best I could come up with was to have one process (rank-0) open a file in sequential mode with
cg_open(cgns_filename, CG_MODE_WRITE, &fn), then the same process writes the base node and the zones. ViaMPI_SendandMPI_Recv, the other processes let rank-0 know how the Zones should be written. So far, rank-0 creates the zones and places the GridCoordinates node under each Zone. After rank-0 does this, it closes the file and then all processes open the file withcgp_file_openwithCG_MODE_MODIFY.The idea being that now that the file is constructed, all processes can reference their respective zone and write their coordinates. I attempted this with
cgp_coord_writebut I get segmentation faults (SIGSEGV). I ran valgrind to attempt to determine where they occur, but it is unable to trace the SIGSEGV to CGNS's API's and blames the underlying HDF5 APIs instead.Any help, insight, or links to better PCGNS resources would be greatly appreciated.
Here pseudo-code of what I have right now:
Beta Was this translation helpful? Give feedback.
All reactions