Connectivity lazy loading, performance and RAM usage

[awoimbee]

Hi,
I have issues with reading old (CGNS <4), large (~30GB) CGNS files:

• it uses an obscene amount of memory
• lots of time is spent on cg_open

This issue is caused by cgi_read_section, which is ran at file-opening-time. If the CGNS file is too old (eg it uses the old NGON/NFACE format), then this function loads the whole connectivity data in RAM and updates it.
Called here:

CGNS/src/cgns_internals.c

Line 595 in 6649d34

if (cgi_read_section(in_link, zone->id, &zone->nsections, &zone->section))

Then functions like cg_poly_elements_read copy the data in memory.

CGNS/src/cgnslib.c

Line 4231 in 6649d34

memcpy(elements, section->connect->data, (size_t)(ElementDataSize*sizeof(cgsize_t)));

Result: when I was using the VTKCGNSReader (directly based on CGIO), I needed less than 100GB of RAM, now that I am using the CGNS MLL I need ~180GB of RAM to open the same ~30GB files.

Would the CGNS team be open to lazy loading ? And to find a way to limit duplicating the connectivity in RAM (although I don't know how to do that without changing the API) ?