cg_open very slow

[CedMoo]

Hello,

I'm currently using CGNS 4.3 and experiment performance issues when calling cg_open. I have some files which takes around 1.5 seconds to open which is way too much for our use case. You might think that this is not that big, but in our specific use case, we can have several thousands of these files to open (such a file is around 90 MB in size). So, going over all the files takes a very long time. What is making things worse is that because of the high number of files, they cannot remain open at the same time, which means that we have to close and reopen them several times, which causes very bad performances.

We actually have some code which is using the low-level CGIO interface to process these files and I wanted to move to the mid-level layer in order to improve performances. There's a lot of searching through the nodes of the different files, which is time consuming. I was expecting the mid-level library to be more efficient (and also would end make our code much more readable) but it turns out that performances are even worse, just by opening the files :(.

Would there be any option to be able to open the files in lazy mode, by loading the data when it is needed? I understand that the data arrays themselves are not loaded (which is understandable) but loading the file structure still takes way too much time.

Thanks.

[jrwrigh]

There's a lot of searching through the nodes of the different files, which is time consuming.

Have you verified that this is the cause of the slow opening? Is the file opening that much slower than your CGIO implementation?

Compiling CGNS with CMAKE_BUILD_TYPE=RelWithDebInfo and then running a small example through perf would help to figure out what exactly is slowing down the function. If it's HDF5 calls, there might be things other than the MLL that can help improve performance (though that's not my area of expertise).

Would there be any option to be able to open the files in lazy mode, by loading the data when it is needed?

How would lazy loading the file structure help here? I assume that after loading the file, some data is read from the file and the mid-level library expects the file structure to be loaded. Having the full file structure loaded will probably necessitate a large change to the code.

I wanted to move to the mid-level layer in order to improve performances.

One of the maintainers can correct me, but the intent of the mid-level library isn't necessarily to get better performance, but to provide an interface that helps ensure that files written are CGNS compliant. Obviously the parallel functions are there for performance, but that's targeting single, large file performance rather than performance with very small files.

[MicK7]

You are right MLL seems to be intended as a "simple" interface to begin writing CGNS file. Thus it sorts zones in alphabetical naming order when opening the file and create a lot of structure in memory. To get a better performance one needs to go at the CGIO level (like what is done in ParaView) and control the parsing of the file.
When not many zones are present and only one big file is read, the MLL and its parallel API should be enough.

[CedMoo]

Hi, thanks for the reply.

Have you verified that this is the cause of the slow opening? Is the file opening that much slower than your CGIO implementation?

Sorry, I meant to say to our current implementation based on CGIO level is slow because of node searching. Since we are not exactly sure about the exact file structure (we need to support files saved with a previous version of CGNS, not only the latest version), we implemented a mechansim to search for nodes based on its label (starting from a certain node, we iteratively look under that node until a certain hierarchy level). This is what is causing the most performance drop in our current implementation.

By moving to the mid-level library, I was hoping to improve the performances since I was expecting that the mid-level library to be more " aware" of the exact file structure and would handle the different file versions. I did not expect however that it would parse the complete file in order to cache the structure in memory.

How would lazy loading the file structure help here?

What I meant here is actually that nothing is loaded from the file when it is opened. Instead, when you call methods like getting the number of zones, or the number of flow solutions inside a specific zone, at this time the file would be accessed to load the requested information. This might sound less effective, but when you require only specific information from the file, this helps on the performances since you only read what you need to read. I know that at this time this will probably be impossible to implement since this require a complete change of the library.

One of the maintainers can correct me, but the intent of the mid-level library isn't necessarily to get better performance, but to provide an interface that helps ensure that files written are CGNS compliant.

Another factor for us was also to be able to write more readable code, since the mid-level library deals directly with CGNS concepts like zones, flow solutions, ...

But ok, as it seems we'll have to stick to the CGIO library then. Is there anyway for us to be able to get an easy overview of the file structure evolution over the different CGNS versions? That could maybe help to write code which requires less scanning through the file and will be more efficient.

[jrwrigh]

How would lazy loading the file structure help here?

What I meant here is actually that nothing is loaded from the file when it is opened. Instead, when you call methods like getting the number of zones, or the number of flow solutions inside a specific zone, at this time the file would be accessed to load the requested information. This might sound less effective

Ah, I see what you're saying. That makes sense. However, the only way that it'd be faster than your CGIO implementation is if the HDF5 standard has an efficient way of reading the entire structure of the file. Otherwise, when you ask, for example, "what are the number of zones?", the code still needs to traverse the graph to determine which nodes are zones and which aren't.

I can't say for sure, but I have a strong suspicion that the slowness of building the internal file structure during a normal cg_open call is dominated by HDF5 library calls rather than the building up of the CGNS internal data structures.

Is there anyway for us to be able to get an easy overview of the file structure evolution over the different CGNS versions?

I'm not sure unfortunately.

[CedMoo]

However, the only way that it'd be faster than your CGIO implementation is if the HDF5 standard has an efficient way of reading the entire structure of the file.

I have quite some experience with the HDF5 API myself and this is the case. Traversing the file hierarchy in HDF5 is very efficient.

I can't say for sure, but I have a strong suspicion that the slowness of building the internal file structure during a normal cg_open call is dominated by HDF5 library calls rather than the building up of the CGNS internal data structures.

I'm pretty sure this is not the case. Like I said, HDF5 is very efficient when traversing the file structure, and 1.5 seconds for loading 90 MB (without loading any of the data itself!) is a terrible performance. So I strongly suspect that the performance issue resides in the mid-level library.

Anyway, changing the way the mid-level library is implemented is probably out of the question, so we'll have to stick to the CGIO level for which we at least have a bit more control on what we can do.

[mennodeij]

Can you share such a file that is giving this performance issues?

[mennodeij]

Try another compressor, like xzip or 7zip maybe

[CedMoo]

I can share the file through our file sharing system but you need to provide your email address, and only you will be able to access the file. Files are too large to upload here (because the file containing the mesh is 120 MB and I would like to share one data file and the mesh at least).

[mennodeij]

I think what's happening here is that each file SideWindowData@... is full of links (more than 400) to the SideWindowData@mesh... file:

HDF5 MotherNode
  +-CGNSLibraryVersion
  +-Base
    +-Export Info:
    +-DES Body
    | +-ZoneType
    | +-StarZoneInfo
    | | +-BoundaryName
    | | +-ContinuumName
    | | +-RegionName
    | | +-RegionType
    | +-GridCoordinates  -> /Base/DES Body/GridCoordinates @ SideWindowData@mesh2.00000e-03.cgns
    | +-DES Body  -> /Base/DES Body/DES Body @ SideWindowData@mesh2.00000e-03.cgns
    | +-Solution00001
    | | +-PointRange
    | | +-GridLocation
    | | +-pressure
    | | | +-DimensionalExponents
    | | +-pressurefake
    | |   +-DimensionalExponents
    | +-ZoneIterativeData
    |   +-FlowSolutionPointers
    +-DES Body 2
    | +-ZoneType
    | +-StarZoneInfo
    | | +-BoundaryName
    | | +-ContinuumName
    | | +-RegionName
    | | +-RegionType
    | +-GridCoordinates  -> /Base/DES Body 2/GridCoordinates @ SideWindowData@mesh2.00000e-03.cgns
    | +-DES Body 2  -> /Base/DES Body 2/DES Body 2 @ SideWindowData@mesh2.00000e-03.cgns
    | +-Solution00001
    | | +-PointRange
    | | +-GridLocation
    | | +-pressure
    | | | +-DimensionalExponents
    | | +-pressurefake
    | |   +-DimensionalExponents
    | +-ZoneIterativeData
    |   +-FlowSolutionPointers
    +-DES Body 2 2
    | +-ZoneType
    | +-StarZoneInfo
    | | +-BoundaryName
    | | +-ContinuumName
    | | +-RegionName
    | | +-RegionType
    | +-GridCoordinates  -> /Base/DES Body 2 2/GridCoordinates @ SideWindowData@mesh2.00000e-03.cgns
    | +-DES Body 2 2  -> /Base/DES Body 2 2/DES Body 2 2 @ SideWindowData@mesh2.00000e-03.cgns
    | +-Solution00001
    | | +-PointRange
    | | +-GridLocation
    | | +-pressure
    | | | +-DimensionalExponents
    | | +-pressurefake
    | |   +-DimensionalExponents
    | +-ZoneIterativeData
    |   +-FlowSolutionPointers
    +-DES Body 2 2 2
    | +-ZoneType
    | +-StarZoneInfo
    | | +-BoundaryName
    | | +-ContinuumName
    | | +-RegionName
    | | +-RegionType
    | +-GridCoordinates  -> /Base/DES Body 2 2 2/GridCoordinates @ SideWindowData@mesh2.00000e-03.cgns
    | +-DES Body 2 2 2  -> /Base/DES Body 2 2 2/DES Body 2 2 2 @ SideWindowData@mesh2.00000e-03.cgns
    | +-Solution00001
    | | +-PointRange

I'm not sure exactly how the mid-level library works, but it possibly opens and closes the referenced file each time.

I've loaded the one of the files into ParaView, then saved from ParaView in CGNS format, so that the saved file no longer has links in it. This cut opening the file from about 5 seconds to less than 1 second (this is on NFS), or from 1.8 seconds to 0.9 seconds on a local disk.
This is still a long loading time. However, if I merge al blocks in ParaView prior to saving the file, the loading time goes down to 0.02 seconds (NFS or local disk).

I'm not sure this helps, but preventing inter file links and merging blocks prior to saving cuts down enormously on loading time.

[CedMoo]

Thanks for the update.

I think what's happening here is that each file SideWindowData@... is full of links (more than 400) to the SideWindowData@mesh...
Yes, that's typically the case with some of the files we have to deal with.

I'm not sure this helps, but preventing inter file links and merging blocks prior to saving cuts down enormously on loading time.
Unfortunately, like I said before, we do not generate those files. These are coming from our customers so we have no control over how these files are created.

In conclusion it means that the mid-level library is not really oriented towards performance. This is the reason why I strongly believe that a lazy loading mechanism would be very useful here (only read and cache what is requested instead of processing the full file).

[MicK7]

Lazy loading should be done on links in this case. It should be handle at the cgio level maybe. Some other implementation of cgns use a link cache table to speed things up.
In ParaView to cope with this kind of issue, there is an option "cache mesh" that will parse the mesh only once to prevent the opening/closing of links when the grid do not change during time.

If you are using the last CGNS 4.4 , may be you should try a bigger value for the option CG_CONFIG_HDF5_ELINK_CACHE_SIZE

[mennodeij]

Unfortunately, like I said before, we do not generate those files. These are coming from our customers so we have no control over how these files are created.

Indeed, but maybe you could add a pre-processing step using ParaView, loading the files (which is fast in PV due to implementation on cgio level), merging blocks using the MergeBlocks filter, then writing back to CGNS prior to starting your processing where now loading with the MLL is performant, especially if your processing is opening the files multiple times.

[CedMoo]

Hi, thanks for your reply. Are you seriously suggesting that we ask our client(s) to open several thousands of files manually, apply some filters and then save the file back :)? And to redo this each time they want to run a new simulation...

No, the conclusion here is that the mid-level library is not really targeted towards efficiency, and should be avoided in this case. This is unfortunate because it would have might the code on our side more robust and more readable (the changes in the file structure would not be handled on our side anymore but inside the library for instance).

[mennodeij]

No, I was more thinking along the lines where you script Paraview in Python with ~20 lines of code with the necessary steps as a first step in your handling of your clients' data, as an alternative to coding the solution in the cgio layer.

I agree with your conclusion that MLL is in this case less performant than it could/should be.