netcdf runtime and postprocessing errors

[rmodrak]

Hi all,

Attached is a benchmark of AxiSEM synthetics against FK synthetics, using the "scak" model from the University of Alaska, Fairbanks. The fits look pretty good!

Getting this result took a lot of debugging. Since AxiSEM is such a useful package for us, if anyone has any advice regarding the following, it'd be really helpful to know.

Thanks,
Ryan

• on our system, instaseis was unable to read netcdf files processed using the field_transform.sh script

• instaseis successfully read netcdf files processed by the field_transform.py script. However, despite trying a variety block_sizes and cache_sizes, the processing was extremely slow (<< 1 MB/s), which led to processing times of several days for a 2 s dominant period simulation

• in short simulations, the AxiSEM solver and mesher ran reliably, but in longer runs or runs with nproc > 4, fatal netcdf runtime errors were a frequent occurrence

• when LOCAL_MAX_COLAT in MESHER/inparm_mesh was uncommented, instaseis was unable to read the resulting netcdf output files

• to compile AxiSEM in debugging mode on our cluster, we had to add -pthread to the gfortran compiler flags in make_axisem.macros

AxiSEM_vs_FK_3s.pdf

[martinvandriel]

Dear Ryan,
thanks for reporting, some thoughts:

on our system, instaseis was unable to read netcdf files processed using the field_transform.sh script

This seems to be an inconsistency in libraries (fortran netcdf vs h5netcdf) that has been around for a while now.

instaseis successfully read netcdf files processed by the field_transform.py script. However, despite trying a variety block_sizes and cache_sizes, the processing was extremely slow (<< 1 MB/s), which led to processing times of several days for a 2 s dominant period simulation

This was a quick hack to solve the first issue, by now I would suggest to use the more versatile tool included in instaseis for repacking (http://instaseis.net/database_repacking.html#repacking-script).

in short simulations, the AxiSEM solver and mesher ran reliably, but in longer runs or runs with nproc > 4, fatal netcdf runtime errors were a frequent occurrence

to compile AxiSEM in debugging mode on our cluster, we had to add -pthread to the gfortran compiler flags in make_axisem.macros

This sounds to me like you are not using the parallel netcdf library, but our own round robin parallelization with threads. Maybe try the parallel netcdf instead (USE_PAR_NETCDF = True in make_axisem.macros)

when LOCAL_MAX_COLAT in MESHER/inparm_mesh was uncommented, instaseis was unable to read the resulting netcdf output files

I think I have been using this feature with instaseis before, but it is definitely not well tested. If you can't get around it, I would need a minimal example to reproduce.

Martin

[rmodrak]

This helps a lot, many thanks.