-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Datatypes changed when using .nc download #89
Comments
That shouldn't happen. But as you'll see below, I can't reproduce the problem. I think the first step to solve this problem is to see what data type is used for that variable in that dataset. So then I made a request for a .nc file with this URL I then used ncdump -h to see what is in the file. It showed (just the part for conductivity_qc): So I have no explanation for why you see float32. Best wishes. |
Hi Bob, Thanks for checking this. You're right, the issue lies in Python xarray's treatment of integers and fill values, not in ERDDAP. I'll use ncdump to check downloaded files in future. I will migrate this issue to erddapy, as the default behavior of xarray is converting integer arrays to float32 at read. |
@BobSimons I'm getting a security policy rejection with your NOAA email 550 5.7.1 unrecognized address. Looks like I may need to be allowlisted . You can contact me at c.rollo@outlook.com |
I saw that problem a few years back but it was gone when the data provider updated their ERDDAP server. I guess that this is a new problem. However, the issue is with xarray and/or maybe the libnetcdf version. So there isn't much we can do in erddapy b/c we just read the downloaded file directly with |
@ocefpaf, can you report the bug to the maintainers of NetCDF4DataStore? |
It appears that variables stored in ERDDAP as integers of various sizes are converted to float32 when exported as netCDF.
Take this example dataset
https://erddap.observations.voiceoftheocean.org/erddap/tabledap/nrt_SEA068_M27.html
The variable
conductivity_qc
is a qc variable that can only have the value of an integer between 1 and 9. As such, we have specified it within ERDDAP as a byte, an 8 bit integerHowever, when this datasets is downloaded as a netCDF this variable, and all others, have been converted to float32. I believe this produces a substantial and avoidable increase in download size.
This issue does not occur appear to occur with export as .csv, as the integers are exported as such, not as floating points.
The text was updated successfully, but these errors were encountered: