ENH: Read new variation of hdf5 file format

[megumillia]

• Py-ART version: 1.17.0 py39ha55989b_0
• Python version: 3.9.18 h1aa4202_0
• Operating System: windows 11

Description

Hello, Im trying to read in some hdf5 data obtained from one brazilian radar. Pyart doesn't seem to be able to read in the file, h5py can. I have tried gamin_hdf5, and posted the error message below at the end after my code.

If it helps I have shared the data (found below)
https://drive.google.com/file/d/1RLOyF2tkuFDd1BbmTUPLiei6mIEkW4m1/view?usp=sharing

Thanks for any advice explanation on why this doesn't work.

What I Did

import pyart
import matplotlib.pyplot as plt
import numpy as np
import cartopy.crs as ccrs
import h5py

radar = pyart.aux_io.read_gamic('radarchapeco.hdf5')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 radar = pyart.aux_io.read_gamic('radarchapeco.hdf5')

File ~\miniconda3\lib\site-packages\pyart\aux_io\gamic_hdf5.py:107, in read_gamic(filename, field_names, additional_metadata, file_field_names, exclude_fields, include_fields, valid_range_from_file, units_from_file, pulse_width, **kwargs)
     97 filemetadata = FileMetadata(
     98     "gamic",
     99     field_names,
   (...)
    103     include_fields,
    104 )
    106 # Open HDF5 file and get handle
--> 107 gfile = GAMICFile(filename)
    109 # verify that all scans are present in file
    110 assert gfile.is_file_complete()

File ~\miniconda3\lib\site-packages\pyart\aux_io\gamicfile.py:40, in GAMICFile.__init__(self, filename)
     38 self._hfile = h5py.File(filename, "r")
     39 self.nsweeps = self._hfile["what"].attrs["sets"]
---> 40 self._scans = ["scan%i" % (i) for i in range(self.nsweeps)]
     41 self.rays_per_sweep = self.how_attrs("ray_count", "int32")
     42 self.total_rays = sum(self.rays_per_sweep)

TypeError: only integer scalar arrays can be converted to a scalar index

[zssherman]

Hello @megumillia ! I can take a look real quick and see. My guess is the attribute nsweep is not an integer

[zssherman]

@megumillia The issue i'm seeing is alot of the attributes in this file are in arrays, instead of strings, int etc. For example your nsweeps instead of 11 is np.ndarray([11]) scan_type is ['ppi'] instead of ppi, etc which py-art does not like as its not expecting arrays for these values

[zssherman]

@megumillia I can try a few things and see if I can get it to read.

[kmuehlbauer]

That's not the only issue here. The timestamps are missing from the ray_header.

[zssherman]

@kmuehlbauer Exactly!

Funny enough, was just dissecting that part of the file now haha:
hfile["scan1"]["ray_header"].attrs.keys()
<KeysViewHDF5 []>

I can think of ways for checking for np.ndarray for the arrays of some of the attributes, but is there anyway around the missing timestamps @kmuehlbauer ? Not too sure how to approach this.

[kmuehlbauer]

We would need to read the sweep timestamps and try to derive ray times from rotation speed.

[zssherman]

Ah gotcha...

[megumillia]

@megumillia I can try a few things and see if I can get it to read.

if you can i will be gratefull, I'm new to this programming and radar data thing.

[zssherman]

@megumillia At the moment I have ways around the np.ndarray parts, but the missing sweep timestamps as @kmuehlbauer mentioned, would require deriving ray times from rotation sweep. That I'm unfamiliar with sadly

[mgrover1]

@megumillia - a new reader would likely need to be written here. If you have the data read in using h5py, I encourage you to check out our notebook example fitting data into the radar object
https://arm-doe.github.io/pyart/notebooks/basic_ingest_using_test_radar_object.html

This would be the best path forward, and once you have a working example, we could work together on a pull request to Py-ART or xradar https://github.com/openradar/xradar (which might be the better option) to have this fully supported. As of now, the file does not conform to any of the readers implemented in Py-ART or xradar.

[megumillia]

@mgrover1 Thanks for the tip and the example link, I'm working on it.

When i finish, i can post here the final code if you want do a pull request to Py-art or xradar.