Parallel cg_boco_write

[jrwrigh]

I'm working on a program that writes out a mesh and boundary conditions to a new CGNS file in parallel. There are dedicated functions for writing node data in parallel, except for cg_boco_write.

First, how difficult would it be to write a cgp_boco_write function? Seems like the underlying infrastructure would be identical to cgp_elements_write_data, but with obvious changes to the node type, etc.

Second, should using cg_boco_write on a file that was opened using cgp_open work correctly? I'm currently running into issues when trying to do this. Specifically, the PointList data is not written out. From cgnsview:

I can't find any example code of using the parallel library and cg_boco_* functions. Most of the parallel library examples do have calls to non-parallel functions (such as cg_base_write), but (and possibly crucially) I don't see any call non-parallel functions which write out array data to nodes.

Is it possible that non-parallel library functions can't write arrays to files opened using cgp_open?

Here's a copy of that cgns file (it's only 32KB, but GitHub will only take a zipped version of the file): chefOut.cgns.gz
I'll see about creating a MWE of the issue tomorrow.

(CCing interested parties @jedbrown @KennethEJansen )

[jrwrigh]

Update: Unfortunately I wasn't able to duplicate the above behavior. I took ptests/test_unstruc_quad.c and added:

  cgsize_t test_points[3] = {1,2,3}, bc_index;
  cg_boco_write(fn, B, Z, "TestBC", CGNS_ENUMV(BCTypeUserDefined), CGNS_ENUMV(PointList), 3, test_points, &bc_index);

right before the file was closed and the ZoneBC node was correctly written to. So my problem lies elsewhere unfortunately.

I'll leave open in case anyone has an idea and if there's an answer to my first question.

[jedbrown]

Just a guess, but I bet that example is only writing from rank 0. A better test would be to have different ranks write different parts.

When I was exploring, I think I expected to use the regular API to ensure metadata was in place, then cgp_elements_write_data or cgp_array_write_data to populate with data. I think the main thing I'm missing in that ad-hoc strategy is how to write the ZoneBC/*/PointList, which is an IndexArray_t and I don't see a way to write those in parallel. One can often order so you only need a PointRange, but that isn't a proper solution. It seems like an oversight in the interface.

[jrwrigh]

Just a guess, but I bet that example is only writing from rank 0. A better test would be to have different ranks write different parts.

I'm pretty sure all ranks are writing out data. Or at least the coordinates being formed are dependent on the rank id.

I think the main thing I'm missing in that ad-hoc strategy is how to write the ZoneBC/*/PointList, which is an IndexArray_t and I don't see a way to write those in parallel.

I don't see a direct way, but the underlying node structure is identical to the element connectivity writing. Only different is the labels put on each node and the data being a vector instead of an array. But otherwise it's just writing out integer values to an array in parallel, so I'd imagine changing some names from the element connectivity writing functions would suffice.

One can often order so you only need a PointRange, but that isn't a proper solution.

I was thinking this too, but the case of multitopology actually makes PointRange not viable assuming MIXED element types are to be avoided.

[ferranj2]

Hi James!

I had (what I hope is the exact same) issue you are facing right now, for months.
And also, I happen to be working on (what I hope is) a similar program to what you're describing.
Not sure if we have the same issues/goals, but I can read in a mesh (for the time being sequentially, but I'm working on a parallel reader), and write it out in parallel.

I tossed the CGNS API out altogether and instead implemented it ("it" as in the CGNS standard directly via HDF5). Granted, for mesh management, boundary condition bookkeeping and all other mesh-related woes, I piggy-backed off the PETSc library. Specifically, I'm using PETSc's DMPlex.

Right now, I can read a Gmsh file (shout out to the PETSc developers who wrote the API for reading Gmsh), call a graph partitioner on it, and write it out as a multi-Zone CGNS file that has ZoneGridConnectivity + ZoneBC (i.e., a partitioned file). I reckon you might not want the partitions?

Here's what I can do via HDF5 + PETSc:

Some CGNSplot screenshots of the mesh shown in that CGNSview:

(a) parMETIS boundaries (I just shaded all the regions):

(b) Sample Boundary Conditions (Below, I select a BC that I prescribed on the back. The BC happened to have been split over two parMETIS partitions).

In my to-do list, I plan to be able to read the CGNS file in parallel, and support parallel writes without partition information.
The Gmsh file I created in Pointwise.

Anyhow, I have source code, albeit, messy as I am removing old functions and adding new ones.
I am a bit of a newbie to software development (I still need to learn how to use Git, the "pulls", "merges", "forks","commits"), but I am happy to share what I've got. I'm trying to write my code in the style of an API so that it could perhaps be used with PETSc.

[jrwrigh]

Hello!

It seems like we're almost working on the inverse; we're writing out a mesh to CGNS (in parallel) so that it can be read by DMPlex! DMPlex will need some updates to the CGNS reader for that to happen, but we're working on that (or rather @jedbrown is).

I reckon you might not want the partitions?

We might want that possibly in the future, but I'm not sure how we're planning on implementing that (by zone or by a id array associated with each element).

I think we'd like to stick in the CGNS library, or contribute to/improve it, but I'll let you know if we want to see your code!

I had (what I hope is the exact same) issue you are facing right now, for months.
Just to clarify, were you having issues with my first question (parallel writing boco information), or the bug I'm seeing where the PointList node isn't getting data written to it?

[ferranj2]

Cool!

It seems like we're almost working on the inverse; we're writing out a mesh to CGNS (in parallel) so that it can be read by DMPlex! DMPlex will need some updates to the CGNS reader for that to happen, but we're working on that (or rather @jedbrown is).

Just out of curiosity, when you say that "DMPlex will read it" do you mean that the CGNS file is reformatted to PETSc's HDF5 format for DMPlex? Or do you mean the assembly of the DMPlex from the information in the CGNS file? If it is the latter, then our work will overlap in terms of the end objective. The only difference would be that I'm not using the actual CGNS API, rather, my own HDF5 version of it.

I think we'd like to stick in the CGNS library, or contribute to/improve it, but I'll let you know if we want to see your code!
Let me know. I'm looking to contribute towards projects.

Just to clarify, were you having issues with my first question (parallel writing boco information), or the bug I'm seeing where the PointList node isn't getting data written to it?

I was having the first issue (of writing the boco in parallel). I didn't run into the PointList one.

[jedbrown]

I'd rather not have PETSc use raw HDF5 to access the CGNS file. (Of course you're free to do whatever.) I think we chatted on the mailing list some time back and we discussed that I'd prefer (for DMPlex) to handle a single zone with multiple topologies and cell labels rather than multi-zone. Some mixed topology meshes of that sort are here (from Pointwise, ANSA, etc.). https://hiliftpw.larc.nasa.gov/Workshop5/grids_downloads_case1.html

when you say that "DMPlex will read it" do you mean that the CGNS file is reformatted to PETSc's HDF5 format for DMPlex? Or do you mean the assembly of the DMPlex from the information in the CGNS file?

I mean parallel loading of CGNS files (DMLoad and VecLoad). The current DMPlex-CGNS serial reader is incomplete and not scalable; I'm working on fixing its functionality and scalability.

[ferranj2]

@jedbrown

I mean parallel loading of CGNS files (DMLoad and VecLoad). The current DMPlex-CGNS serial reader is incomplete and not scalable; I'm working on fixing its functionality and scalability.

Gotcha. That's sort of what I meant by "assembly of the DMPlex from the information in the CGNS file". Right now I'm more in the DMView() territory, but soon I will get into the DMLoad() territory.

Yeah, I recall from looking at the source code for DMPlexCreateCGNSFromFile() that it has rank 0 do everything.

Not sure if this is relevant to you or your group, but my approach (in trying to make the reader or DMLoad) parallel is to call DMPlexBuildFromCellListParallel() and DMPlexBuildCoordinatesFromCellListParallel() with arrays that I read off HDF5 (which you would read from CGNS). The former, "BuildFromCellList" needs an upgrade because it is hard coded for meshes of a single polytope type. I am yet to conceptually figure out how to read the ZoneBC in parallel.

[brtnfld]

It would be straightforward to implement a parallel version of cg_boco_write. Do you have, for example, a zone that is partitioned among multiple ranks? Do you have an example of how you are calling cg_boco_write? The issue with calling the serial version in parallel would be that all the ranks need to call it with the same input parameters. For example, you could not have the bocname be different between ranks. HDF5 requires file structure elements (group, dataset, etc..) to be a collective operation, i.e., you can't have one rank create a group the other ranks don't know about, and they all must create objects with the same size dimensions. If you meet that criterion, then what would happen is all the ranks will write the same data to the file. If they have different data between the ranks, the final data would be whichever rank wrote last.

[jrwrigh]

It would be straightforward to implement a parallel version of cg_boco_write.

It'd be equivalent of the cgp_element_write* functions, but with different names, node types, etc.?

Do you have, for example, a zone that is partitioned among multiple ranks?

That is the future plan, yes. Note that the above behavior is observed for a serial process, but using the parallel interface. Which is the part that is quite confusing to me.

[brtnfld]

If all the ranks call cg_boco_write with the same parameters, it should work so it might be a bug. Does it work correctly when using one rank?

Correct, it would have a similar API as the other cpg_*write/read APIs.

Remember, if you have a bunch of small BC datasets each rank contributes to, the performance might be poor. Parallel I/O performance is best with large data with the fewest IO calls.

[jedbrown]

For scale-resolving CFD, a billion element mesh might have boundaries of size only a couple million and writing from rank 0 is acceptable (even if slightly awkward to MPI_Allgatherv the ~100 MB). This gets a bit more painful as the problem size grows and you work with complex geometry. In structures, especially thin inflatable structures, the number of boundary elements on which BCs must be applied is similar to the number of volume elements so a billion element problem will have boundary sets of similar size and we surely don't want to force those tens of GB of data to funnel through rank 0. I'm happy with the collective semantics.

[KennethEJansen]

Scott makes a good point. Ultimately, we have 2 options.

1. current path which is to compute the bc data in its current, parallel, distributed form and use a cgp_writeBoco function when it gets developed.
2. Collect that distributed data to rank0 and use the existing serial BC writer.

On the reader side within PETSc we presumably have the same two options independent of the choice on the writer side.

My guess is that option is better even though I initially pushed @jrwrigh towards 1 because I get Scott’s point that performance wise we may not see 1 being faster than 2 anyway.

[KennethEJansen]

In light of @jedbrown point (which I did not see) modify my comment above to say, path 2 is ok near term but we need path 1 to work for the extreme scale meshes.

[brtnfld]

If you have enough data to keep a parallel file system happy, option 1 is the way to go. With option two, you can go surprisingly big in sending data and still get good performance compared to having all the ranks doing small I/O. We added an optimization in HDF5 (mainly for CGNS) that if all the ranks are writing the same data (H5S_ALL is being used, < 2GiB), only rank 0 will write the data, or rank 0 will read the data and bcast it.

But BC datasets, by default in CGNS, are compact datasets, so it already has this behavior (this was added to later versions of CGNS)

Also, you could use the HDF5 multi-dataset APIs to combine all the BC datasets, which would help by combining them into a single MPI-derived type and writing them all at once.

[jedbrown]

For unstructured BCs, the commercial meshers create different sections for each named BC, but I like the parallel IO semantics better of one section for all the tri (and all the quad elements if applicable), then let the ZoneBC identify the element subsets.

If user code doing an MPI_Gatherv to rank 0 for serial writing is significantly faster than collective MPI-IO, that sounds like a performance bug in MPI-IO. If we have GBs of distributed array data, I'd rather use a collective interface. Is this possible now (perhaps using lower level interfaces) or does a new cgp_boco_write need to be added?

[brtnfld]

Is this possible now (perhaps using lower level interfaces) or does a new cgp_boco_write need to be added?

cg_boco_write can not do collective (or independent) IO as it always writes to the entire dataset. If it is called by all the ranks with the same parameters, then only rank 0 will do the actual writing.

[jedbrown]

Sounds like we should gather to rank 0 and call cg_boco_write for now, and request a parallel version (or some guidance on how to add it -- cgp_section_write and cgp_elements_write_data are perhaps an adequate model to compare to).

[KennethEJansen]

As an alternative to writing code we hope to replace, should we consider using the deprecated route inferring boundary conditions from the boundary element list and a single integer tag of the same length as the boundary element list that tells us the model face each boundary element is on?

I have not checked to see if there is a parallel writer (and reader) for that model tag number (or more generally boundary element data) but if there is then we would basically be done as soon as @jrwrigh figures out how to write it because I already have it extracted from the SCOREC/core mesh database and ready to write and confirmed the other parts are writing in parallel correctly. This is our last step on the writer side and that enables @jedbrown to get the parallel reader developed.

Again, this is a short term solution and I am sure we would want to use a cgp_boco_write/read and could help in that development but I am thinking, for very large meshes, this deprecated route could be the fastest until that is ready. I am further thinking that, while I trust that @brtnfld knows the performance tradeoffs and the low to moderate difficulty of that development, something that is ready today, even if deprecated, takes the time pressure off of the non-deprecated development.

As I look a bit into this idea, I am unsure if it is supported. Does CGNS support cg_sol_ptset_write being pointed at cell centers with a PointList range limited to the boundary elements? I see that for serial. I have not yet figured out whether cgp_field_general_write_data can do the same?

@jedbrown if it does, would PETSC be able to read the same?

[KennethEJansen]

So I was not able to use the field based function since they can only be attached to "volume" elements or nodes (at least I think that is the case). However, I was able to attach that as "user" data.

(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mdsAllHex/Chef/1-1-Chef (CGNS_tests)$ cgnslist chefOut.cgns 
HDF5 MotherNode
  +-CGNSLibraryVersion
  +-Base
    +-Zone
      +-ZoneType
      +-GridCoordinates
      | +-CoordinateX
      | +-CoordinateY
      | +-CoordinateZ
      +-Hex
      | +-ElementRange
      | +-ElementConnectivity
      +-RankCellOwner
      | +-GridLocation
      | +-RankOfWriter
      +-Quad
      | +-ElementRange
      | +-ElementConnectivity
      +-User Data
        +-srfID

@jedbrown please comment as to whether you think you will be able to do a parallel read on srfID [numBelTp] entries with alignment with ElementConnectivity. I tried to put it under the Quad node but this is not allowed apparently

[process 0]:User defined DataArray_t node not supported under 'Elements_t' type node

While this is a deprecated approach, perhaps it will give us parallel r/w performance until cgp_boco_read/write is finished.

[jedbrown]

We could do a parallel read of that, but I think we should just use the supported method with MPI_Gatherv and MPI_Scatterv. I think the surfaces of your meshes only have a few million elements (not billions) so this won't be a big deal. I'd rather minimize the use of ad-hoc conventions because PETSc users will also need to read CGNS meshes produced by Pointwise, ANSA, etc., so it's not like I can avoid writing code by assuming a bespoke convention.

[KennethEJansen]

Point taken and understood.

Correct about a few million elements on a given surface and that is at lower order. For Q3 it will be significantly less.

[jrwrigh]

Regarding the PointList error I was seeing earlier, after upgrading from CGNS 4.2.0 to CGNS 4.4.0. Both were built using spack, so I'll give the full dependency tree for the one that worked and the one that didn't (found using spack find -d --variants):

CGNS that worked:

cgns@4.4.0~base_scope~fortran+hdf5+int64~ipo~legacy~mem_debug+mpi+scoping+shared~static~testing~tools build_system=cmake build_type=RelWithDebInfo generator=make
    cmake@3.26.3~doc+ncurses+ownlibs~qt build_system=generic build_type=Release
        curl@8.1.2~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl
            nghttp2@1.52.0 build_system=autotools
            openssl@3.1.2~docs~shared build_system=generic certs=mozilla
                ca-certificates-mozilla@2023-05-30 build_system=generic
        ncurses@6.4~symlinks+termlib abi=none build_system=autotools
        zlib@1.2.13+optimize+pic+shared build_system=makefile
    gmake@4.4.1~guile build_system=autotools
    hdf5@1.14.1-2~cxx~fortran~hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make
        pkgconf@1.9.5 build_system=autotools
    openmpi@4.1.5~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+romio+rsh~singul
arity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none
        hwloc@2.9.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static
            libpciaccess@0.17 build_system=autotools
                util-macros@1.19.3 build_system=autotools
            libxml2@2.10.3~python build_system=autotools
                libiconv@1.17 build_system=autotools libs=shared,static
                xz@5.4.1~pic build_system=autotools libs=shared,static
        numactl@2.0.14 build_system=autotools patches=4e1d78c,62fc8a8,ff37630
            autoconf@2.69 build_system=autotools patches=35c4492,7793209,a49dd5b
            automake@1.16.5 build_system=autotools
            libtool@2.4.7 build_system=autotools
            m4@1.4.19+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7
                diffutils@3.9 build_system=autotools
                libsigsegv@2.14 build_system=autotools
        openssh@9.3p1+gssapi build_system=autotools
            krb5@1.20.1+shared build_system=autotools
                bison@3.8.2 build_system=autotools
                findutils@4.9.0 build_system=autotools patches=440b954
                gettext@0.21.1+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools
                    tar@1.34 build_system=autotools zip=pigz
                        pigz@2.7 build_system=makefile
                        zstd@1.5.5+programs build_system=makefile compression=none libs=shared,static
            libedit@3.1-20210216 build_system=autotools
            libxcrypt@4.4.35~obsolete_api build_system=autotools patches=4885da3
        perl@5.38.0+cpanm+opcode+open+shared+threads build_system=generic
            berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc
            bzip2@1.0.8~debug~pic+shared build_system=generic
            gdbm@1.23 build_system=autotools
                readline@8.2 build_system=autotools patches=bbf97f1
        pmix@4.2.4~docs+pmi_backwards_compatibility~python~restful build_system=autotools
            libevent@2.1.12+openssl build_system=autotools

CGNS that didn't work:

cgns@4.2.0~base_scope~fortran+hdf5+int64~ipo~legacy~mem_debug+mpi+scoping+shared~static~testing build_type=RelWithDebInfo
    hdf5@1.10.7~cxx~fortran~hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo
        openmpi@4.1.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multipl
e+vt+wrapper-rpath fabrics=none schedulers=none
            hwloc@2.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
                libpciaccess@0.16
                libxml2@2.9.12~python
                    libiconv@1.16 libs=shared,static
                    xz@5.2.5~pic libs=shared,static
                    zlib@1.2.11+optimize+pic+shared
                ncurses@6.2~symlinks+termlib abi=none
            libevent@2.1.12+openssl
                openssl@1.1.1l~docs certs=system
            numactl@2.0.14 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c00
6,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296
            openssh@8.7p1
                libedit@3.1-20210216
        pkgconf@1.8.0

So there are a few packages that have newer versions. Namely:
HDF5 1.10.7 -> 1.14.1
OpenMPI 4.1.1 -> 4.1.5

Regardless, question 2 (the actual bug with PointList), that's now resolved. I'll leave the discussion open regarding the parallelization of cg_boco_write.

[KennethEJansen]

Hey @brtnfld
How concerned should I be about this valgrind output that suggest pwritev has quite a few issues for my exec. Pasting just 1 of about 50. (let me know if you want all of them)

   9 ==3802== Syscall param pwritev(vector[8]) points to uninitialised byte(s)
  10 ==3802==    at 0xB3AB43B: pwritev (pwritev.c:47)
  11 ==3802==    by 0xA3E1B95: mca_fbtl_posix_pwritev (in /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/lib/libmpi.so.40.30.1)
  12 ==3802==    by 0xA3767C8: mca_common_ompio_file_write (in /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/lib/libmpi.so.40.30.1)
  13 ==3802==    by 0xA376258: mca_common_ompio_file_write_at_all (in /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/lib/libmpi.so.40.30.1)
  14 ==3802==    by 0xA431896: mca_io_ompio_file_write_at_all (in /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/lib/libmpi.so.40.30.1)
  15 ==3802==    by 0xA34DCE7: PMPI_File_write_at_all (in /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/lib/libmpi.so.40.30.1)
  16 ==3802==    by 0x9C1CEAB: H5FD_mpio_write (H5FDmpio.c:1636)
  17 ==3802==    by 0x9C199CD: H5FD_write (H5FDint.c:248)
  18 ==3802==    by 0x9BF8CF3: H5F__accum_write (H5Faccum.c:823)
  19 ==3802==    by 0x9CF4B92: H5PB_write (H5PB.c:1031)
  20 ==3802==    by 0x9C033BA: H5F_block_write (H5Fio.c:160)
  21 ==3802==    by 0x9B930A9: H5C__collective_write (H5Cmpio.c:1109)
  22 ==3802==    by 0x9B930A9: H5C_apply_candidate_list (H5Cmpio.c:402)
  23 ==3802==  Address 0x10036ec8 is 56 bytes inside a block of size 160 alloc'd
  24 ==3802==    at 0x4C40E2B: malloc (vg_replace_malloc.c:380)
  25 ==3802==    by 0x9B837C1: H5C__flush_single_entry (H5C.c:6886)
  26 ==3802==    by 0x9B92D49: H5C__flush_candidates_in_ring (H5Cmpio.c:1421)
  27 ==3802==    by 0x9B92D49: H5C__flush_candidate_entries (H5Cmpio.c:1242)
  28 ==3802==    by 0x9B92D49: H5C_apply_candidate_list (H5Cmpio.c:393)
  29 ==3802==    by 0x9B65732: H5AC__rsp__dist_md_write__flush (H5ACmpio.c:1707)
  30 ==3802==    by 0x9B65732: H5AC__run_sync_point (H5ACmpio.c:2181)
  31 ==3802==    by 0x9B65732: H5AC__run_sync_point (H5ACmpio.c:2110)
  32 ==3802==    by 0x9B6627A: H5AC__flush_entries (H5ACmpio.c:2324)
  33 ==3802==    by 0x9B5F64C: H5AC_flush (H5AC.c:740)
  34 ==3802==    by 0x9BFC6B0: H5F__flush_phase2.part.0 (H5Fint.c:1988)
  35 ==3802==    by 0x9BFE82E: H5F__flush_phase2 (H5Fint.c:1977)
  36 ==3802==    by 0x9BFE82E: H5F__dest (H5Fint.c:1255)
  37 ==3802==    by 0x9BFEECA: H5F_try_close (H5Fint.c:2345)
  38 ==3802==    by 0x9BFF1DB: H5F__close_cb (H5Fint.c:2172)
  39 ==3802==    by 0x9C74937: H5I_dec_ref (H5I.c:1261)
  40 ==3802==    by 0x9C74937: H5I_dec_ref (H5I.c:1226)
  41 ==3802==    by 0x9C749F6: H5I_dec_app_ref (H5I.c:1306)
  42 ==3802==    by 0x9C749F6: H5I_dec_app_ref (H5I.c:1295)

[brtnfld]

I don't think it is an issue. In the past, we attributed that to mpi.

[KennethEJansen]

Thanks for the response. I am flying a little blind here as we don't have a reader yet for this file and I am trying to do all I can to verify correctness. Note I did use the "known" openmpi suppression e.g.,

mpirun -np 2 valgrind --log-file=vg.%p --suppressions=/usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/share/openmpi/openmpi-valgrind.supp --suppressions=/usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/share/pmix/pmix-valgrind.supp /projects/tools/SCOREC-core/buildCGNS_OneBase/test/chef 2>&1 |tee -a  val10.Log

but I can imagine these files may not cover everything. I will see if @cwsmith (controller of PRs on SCOREC/core) knows of any other suppression files for MPIO and/or HDF5 which might be needed to get a clean bill of health from valgrind which is always nice to have.

[jedbrown]

In case someone finds this thread and wants to know what came of it, #730 provides this feature.

[jrwrigh]

Huh. I thought the "close" tags on the PR would work for discussions.