CDF-5 support?

[rschmunk]

Does netCDF-Java support NC files in the CDF-5 format?

A user pinged me because the developers of the regional climate model he uses are apparently updating their code to produce data output in CDF-5 format. Panoply (using a NJ 5.5.4 snapshot) cannot open the file and the stack trace I get checking into this says

java.io.IOException: java.io.IOException: Cant read file:/Users/rbs/Downloads/jays_file_DOMAIN000.nc: not a valid CDM file. at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:279) at ucar.nc2.dataset.NetcdfDatasets.openProtocolOrFile(NetcdfDatasets.java:455) at ucar.nc2.dataset.NetcdfDatasets.openDataset(NetcdfDatasets.java:152) at ucar.nc2.dataset.NetcdfDatasets.acquireDataset(NetcdfDatasets.java:280) at ucar.nc2.dataset.NetcdfDatasets.acquireDataset(NetcdfDatasets.java:257) at gov.nasa.giss.data.nc.NcDataset.acquireNjDataset(NcDataset.java:668)

I also get the java.io.IOException trying to open the file using the latest IDV.

[haileyajohnson]

I'm not sure, I'll look into it and get back to you :)

[haileyajohnson]

It looks like we don't natively support reading CDF5, but it is implemented through the c library, i.e. it should work with the netcdf4 module and a copy of the netcdf-c library installed. Which would explain why it doesn't work in the IDV and Panoply. I'm not sure what the solution should be yet.

[rschmunk]

@haileyajohnson, So theoretically, if the user had the netCDF-C library installed, netCDF-Java would be able to open a CDF-5 file if RuntimeConfigParser was told to read in a properly configured $HOME/.unidata/nj22Config.xml ?

[haileyajohnson]

With the "theoretically" doing a lot of work there, I believe so :)

[JohnLCaron]

Do we have any sample CDF-5 files?

[rschmunk]

@JohnLCaron, The one I was provided is available here.

The user also provided a sample, equivalent NC4 dataset, and it appears that for multiple variables, the data area is a swath across the North Pacific, including part of Alaska.

[JohnLCaron]

Im looking over the pnetcdf format to remember why we didnt implement it. It seems that the main problem is that dimension lengths are int64s, whereas in the classic model they are int32s. Its not a big deal to widen, but its a breaking change to the API in lots of places, since indices have to also be int64.

Im looking at the netcdf4 api to understand how it supports pnetcdf. It is using size_t everywhere for dimension lengths and indexes, which i didnt realize until now.

so, we could switch to using int64 for dimensions and indices in netcdf-java, but it would be a breaking change.

Reading now using the netcdf-C JNA interface must be truncating everything to 32 bits, so it will only work on variables with dimensions of length < 2^31.

It would be simple to extend the existing code to support pnetcdf4 (using long offsets like CDF2) with the same restrictions (indiviidual dimension lengths < 2^31 (maybe <2^32). Not a problem for the variable to have more than that total nelems, as long as each dimension length was an int32.

Note one can only ask to read total nelems < 2^31, since thats as big as Java arrays get. Ive always thought it was ok to require the user to break their reads up into nelems < 2^31. To lift that restriction would be significantly more work.

[rschmunk]

@JohnLCaron, FWIW, I have Panoply coded to "chunk" its requests if it determines a complete 2D slice out of a variable would contain too many elements. The max chunk size varies depending on the heap that's been allocated, but it's never more than 36M elements. I first implemented this scheme in an early form years ago when max physical memory on personal computers was rarely more than 8GB.

[JohnLCaron]

Yeah, any non-toy program has to break things up to fit into physical memory.

Im looking at the file you sent to understand the format better. They are using int64 for all kinds of things, including eg the number of dimensions, attributes and variables. I hope no one has more than a few hundred of any of those. My linear searches will get really slow. Might even have some O(n^2) problems.

So maybe they dont really expect any one dimension to exceed 2^32, and we could just barf if we encoutered that. Maybe we should ask pnetcdf people if that happens much.

Under that assumption, the only important improvement of CDF5 over CDF2 is 1) number of records not limited to 2^32, and 2) the total size in bytes of a variable also not limited.

[JohnLCaron]

And number of records == unlimited dimension length, so then the only improvement is total size in bytes of a variable. Unless you decide to make dim lengths and indexes int64 in your whole API.

[rschmunk]

FWIW: The dataset provider states that, "These model output are from the RegCM5 regional model (originally from NCAR, but now managed in Trieste, Italy)." But from what he said, I don't think all of what they're doing is set in stone yet. Any guidance I can pass along would probably be helpful.

[JohnLCaron]

well, the right thing to do is to use int64s, but thats not going to happen in netcdf-java ver5.

Ask them if they anticipate any dimension lengths (including unlimited) being larger than 2^31-1 (not 2^32-1, because dimensions are signed ints). If not, we could support that.