Support nczarr conversion.

[hdfeos]

netCDF-4.8.0 supports nczarr but NCO is not.

$ncks -4 "file:///home/ubuntu/src/netcdf-c-4.8.0/nczarr_test/tmp_ndims.file#mode=nczarr,file" test.h5
ncks: ERROR file file:///home/ubuntu/src/netcdf-c-4.8.0/nczarr_test/tmp_ndims.file#mode=nczarr,\
file not found. It does not exist on the local filesystem, nor does it match remote filename pa\
tterns (e.g., http://foo or foo.bar.edu:file).

[czender]

Correct. I have a proposal pending with NASA to add that support. They rejected the first version last year, let's hope this one is more successful.

[czender]

FYI the current snapshot of NCO has nascent support for NCZarr. Most operations work for most operators using the file:/// protocol. S3 support will take longer. Testing would be very welcome. The API has not changed, just sprinkle NCZarr-parseable file/bucketnames where appropriate, e.g.:

zender@spectral:~$ ncks "file://${HOME}/zarr_out#mode=nczarr,zarr"
netcdf zarr_out {
  dimensions:
    dmn = 2 ;

  variables:
    int var(dmn) ;
      var:purpose = "var" ;
      var:_FillValue = -32767s ;
      var:test = 2000s ;

  // global attributes:
    :history = "Mon Jul 18 06:09:22 2022: ncatted -a test,var,c,d,2.0e3 -O file:///Users/zender/zarr#mode=nczarr,zarr file:///Users/zender/zarr_out#mode=nczarr,zarr" ;
    :NCO = "netCDF Operators version 5.1.1-alpha04 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;


  data:
    var = 1, _ ;

} // group /

[hdfeos]

Hi, ncks fails to create Zarr files: $ ncks SMAP_L3_SM_P_20150406_R14010_001.h5 "file://foo#mode=nczarr,file" nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_def_var_deflate() nco_err_exit(): ERROR Error code is -136. Translation into English with nc_strerror(-136) is "NetCDF: Filter error: undefined filter encountered" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) $ ncks ATL08_20181014084920_02400109_003_01.h5 "file://foo#mode=nczarr,file" ERROR: nco_def_dim() cannot define dimension "phony_dim_7" with illegal size = 0L nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_def_dim() nco_err_exit(): ERROR Error code is -63. Translation into English with nc_strerror(-63) is "NetCDF: Invalid dimension size" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) [1] https://github.com/hyoklee/kerchunk/blob/main/data/SMAP_L3_SM_P_20150406_R14010_001.h5 [2] https://github.com/hyoklee/kerchunk/blob/main/data/ATL08_20181014084920_02400109_003_01.h5

…

On Thu, Jul 28, 2022 at 10:18 PM Charlie Zender ***@***.***> wrote: FYI the current snapshot of NCO has nascent support for NCZarr. Most operations work for most operators using the file:/// protocol. S3 support will take longer. Testing would be very welcome. The API has not changed, just sprinkle NCZarr-parseable file/bucketnames where appropriate, e.g.: ***@***.***:~$ ncks "file://${HOME}/zarr_out#mode=nczarr,zarr" netcdf zarr_out { dimensions: dmn = 2 ; variables: int var(dmn) ; var:purpose = "var" ; var:_FillValue = -32767s ; var:test = 2000s ; // global attributes: :history = "Mon Jul 18 06:09:22 2022: ncatted -a test,var,c,d,2.0e3 -O file:///Users/zender/zarr#mode=nczarr,zarr file:///Users/zender/zarr_out#mode=nczarr,zarr" ; :NCO = "netCDF Operators version 5.1.1-alpha04 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ; data: var = 1, _ ; } // group / — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

-- - HDF-EOS Tools and Information Center Team

[czender]

Thanks for that report. NCO breaks while trying to copy the string variables. Not sure why, possibly because they are record variables that NCZarr does not yet support? In any case, to workaround, omit them:

zender@spectral:~$ ncks -O -x -v tb_time.? /Users/zender/data/hdf/SMAP_L3_SM_P_20150406_R14010_001.h5 "file:///Users/zender/foo#mode=nczarr,file"
zender@spectral:~$