Releases: Deltares/hydromt
v0.9.4
This release fixes a performance regression when reading geometry masks, relaxed warnings for empty raster datasets and updated the documentation of the new hydromt commands.
Fixed
v0.9.3
v0.9.3 (2024-02-08)
This release fixes several bugs. Most notably the NoDataSrategy
is available in much more data reading methods so plugins can use it more directly. Additionally there are some bug fixes relating to reading shapefiles and reading COGs.
Added
- Test script for testing predefined catalogs locally. (#735)
- Option to write a data catalog to a csv file (#425)
Fixed
- Reading Vector formats that consist of more than one file via geopandas. (#691)
- Handle NoDataStrategy consistently when reading data in adapters (#738)
- add option to ignore empty data sets when exporting data (#743)
- Fix bug in
raster._check_dimensions
for datasets with multiple variables with varying dimension size (#761) - Fix bug when reading COGs at requested zoom level (#758)
v0.9.2
v0.9.2 (2024-01-09)
This release adds additional bug fixes for the meridian offset functinality, and improvements to the new CLI commands.
Added
- Export CLI now also accepts time tuples (#660)
- New stats.skills VE and RSR (#666)
- Check CLI command can now validate bbox and geom regions (#664)
Changed
- Export CLI now uses '-s' for source, '-t' for time and '-i' for config. (#660)
Fixed
- Double reading of model components when in appending mode. (#695)
- Removed deprecated entrypoints library. (#676)
- Bug in
raster.set_crs
if input_crs is of type CRS. (#659) - Export CLI now actually parses provided geoms. (#660)
- Bug in stats.skills for computation of pbias and MSE / RMSE. (#666)
Model.write_geoms
ow has an option to write GeoJSON coordinates in WGS84 if specified (#510)- Fix bug with lazy spatial_ref coordinate (#682)
- Bug in gis_utils.meridian_offset. (#692)
v0.9.1
- Pyogrio io by @Jaapel in #583
- fix some docs issues by @savente93 in #619
- To stac by @savente93 in #617
- Implement from stac for data catalog by @savente93 in #625
- Transform boundary fix by @Tjalling-dejong in #629
- Implement export data via cli by @savente93 in #627
- Add more diagrams to the documentation for users. by @savente93 in #631
- Add Pixi as a task runner by @savente93 in #634
- Add validation methods for data catalog by @savente93 in #632
- fix zoom level bugs by @DirkEilander in #642
- Model config validation by @savente93 in #643
- Fix meridian offset by @DirkEilander in #649
v0.9.0
This release contains several new features, here we highlight a few:
- Support in the DataCatalog for data sources from different providers or versions with better support for cloud and http data.
- Developers documentation to start your own plugin and accompanying template.
- Support multigrids in meshmodel (with example) and improved implementation VectorModel (was LumpedModel)
- Support for reading overviews (zoom levels) of Cloud Optimized GeoTIFFs (COGs).
Added
Documentation
- docs now include a dropdown for selecting older versions of the docs. (PR #457)
- docs include a new example for MeshModel. (PR #595)
- Added documentation for how to start your own plugin (PR #446)
Data
- Support for loading the same data source but from different providers (e.g., local & aws) and versions (PR #438)
- Add support for reading and writing tabular data in
parquet
format. (PR #445) - Add support for reading model configs in
TOML
format. (PR #444) - add
open_mfcsv
function inio
module for combining multiple CSV files into one dataset. (PR #486) - Adapters can now clip data that is passed through a python object the same way as through the data catalog. (PR #481)
- Relevant data adapters now have functionality for reporting and detecting the spatial and temporal extent they cover (PR #503)
- Data catalogs have a
hydromt_version
meta key that is used to determine compatibility between the catalog and the installed hydromt version. (PR #506) - Allow the root of a data catalog to point to an archive, this will be extracted to the ~/.hydromt_data folder. (PR #512)
- Support for reading overviews from (Cloud Optimized) GeoTIFFs using the zoom_level argument of
DataCatalog.get_rasterdataset
. (PR #514) - Support for http and other filesystems in path of data source (PR #515).
Model
- new
force-overwrite
option inhydromt update
CLI to force overwritting updated netcdf files. (PR #460) - Model objects now have a _MODEL_VERSION attribute that plugins can use for compatibility purposes (PR # 495)
set_forcing
can now add pandas.DataFrame object to forcing. (PR #534)
Raster
- Model class now has methods for getting, setting, reading and writing arbitrary tabular data. (PR #502)
- Support for writing overviews to (Cloud Optimized) GeoTIFFs in the
raster.to_raster
method. (PR #514) - New raster method
to_slippy_tiles
: tiling of a raster dataset according to the slippy tile structure for e.g., webviewers (PR #440).
Changed
Model
- Updated
MeshModel
and related methods to support multigrids instead of one single 2D grid. (PR #412) - Renamed
LumpedModel.response_units
toVectorModel.vector
and updated the base set, read, write methods. (#531) - possibility to
load
the data in the model read_ functions for netcdf files (default for read_grid in r+ mode). (PR #460) - Internal model components (e.g.
Models._maps
,GridModel._grid``) are now initialized with None and should not be accessed directly, call the corresponding model property (e.g.
Model.maps,
GridModel.grid`) instead. (PR #473) setup_mesh2d_from_rasterdataset
andsetup_mesh2d_from_raster_reclass
now use xugrid Regridder methods. (PR #535)- Use the Model.data_catalog to read the model region if defined by a geom or grid. (PR #479)
Vector
vector.GeoDataset.from_gdf
can use the gdf columns as data_vars instead of external xarray. (PR #412)
Fixed
- when a model component (eg maps, forcing, grid) is updated using the set_ methods, it will first be read to avoid loosing data. (PR #460)
- open_geodataset with driver vector also works for other geometry type than points. (PR #509)
- overwrite model in update mode. (PR #534)
- fix stats.extremes methods for (dask) 3D arrays. (PR #505)
- raster gives better error on incompatible nodata (PR #544)
Deprecated
- the dependencies
pcraster
andpygeos
are no longer used and were removed. (PR #467)
v0.8.0
This release contains several new features, including extreme value analysis, new generic methods for the GridModel
class, setting variable attributes like units through the data catalog, and the ability to detect compatibility issues between Datacatalog and HydroMT versions. It also includes a minor breaking change since now geometry masks are only set if the mask
in raster.clip_geom
is set to True
to improve memory usage.
What's Changed
Added
- Support for unit attributes for all data types in the DataCatalog. PR #334
- Data catalog can now handle specification of HydroMT version
- New generic methods for
GridModel
:setup_grid
,setup_grid_from_constant
,setup_grid_from_rasterdataset
,setup_grid_from_raster_reclass
,setup_grid_from_geodataframe
. PR #333 - New
grid
workflow methods to support the setup methods inGridModel
:grid_from_constant
,grid_from_rasterdataset
,grid_from_raster_reclass
,grid_from_geodataframe
.
PR #333 - New raster method
rasterize_geometry
. - New extreme valua analysis and design event (creation hydrographs) methods in stats submodule.
Note that these methods are experimental and may be moved elsewhere / change in signature. PR #85
Changed
- Arguments to drivers in data catalog files and the
DataCatalog.get_
methods should now explicitly be called driver_kwargs instead of kwargs. PR #334 - New geom_type argument in
RasterDataArray.vector_grid
to specify the geometry type {'polygon', 'line', 'points'} of the vector grid. PR #351 - Added extrapolate option to
raster.interpolate_na
method. PR #348 - Name of methods
setup_maps_from_raster
andsetup_mesh_from_raster
tosetup_maps_from_rasterdataset
andsetup_mesh_from_rasterdataset
. PR #333 - Add rename argument to
setup_*_from_rasterdataset
,setup_*_from_raster_reclass
to maps and mesh for consistency with grid. PR #333 - Introduced different merge options in
GeoDataset.from_gdf
andGeoDataFrame.from_gdf
. PR #441 DataCatalog.get_rasterdataset
always uses bbox to clip raster data. PR #434raster.clip_geom
only set a geometry mask if the mask argument is true to avoid memory issues. PR #434raster.clip_mask
interface and behavior changed to be consistent withraster.clip_geom
. PR #318
Fixed
- Order of renaming variables in
DataCatalog.get_rasterdataset
for x,y dimensions. PR #324 - fix bug in
get_basin_geometry
for region kind 'subbasin' if no stream or outlet option is specified. - fix use of Path objects in
DataCatalog.from_dict
. PR #429 raster.reproject_like
first clips the data to the target extent before reprojecting. PR #434
v0.7.1
This release contains several small updates of the code.
Most prominently is the support for yml configuration files.
Full list of PRs (auto generated)
- DOC: JOSS paper by @DirkEilander in #303
- ENH: Create issue templates by @alimeshgi in #308
- BUG: fix error if crs property of Model.grid is None by @DirkEilander in #294
- ENH: support exporting individual variables in DataCatalog.export_data by @DirkEilander in #302
- ENH: support YAML configuration files by @Tjalling-dejong in #292
- ENH: support data like objects in DataCatalog.get methods by @DirkEilander in #313
- BUG: small fix parsing paths in yml by @DirkEilander in #315
- BUG: small bugs related to new yml ini files by @DirkEilander in #319
New Contributors
- @alimeshgi made their first contribution in #308
Full Changelog: v0.7.0...v0.7.1
v0.7.0
This release contains several major updates of the code. These following updates might require small changes to your code:
- Most noticeable is the change in the
hydromt build
CLI, where made the region argument optional and deprecated the resolution option. Futhermore, the user has to force existing folders to be overwritten when building new models. - We also did a major overhaul of the
GeoDataset
and the associated.vector
assessor to support any type of vector geometries (before only points).
More new features, including support for rotated grids, new cloud data catalogs and (caching of) tiled raster datasets and more details are listed below.
Changed
- Removed resolution ('-r', '--res') from the hydromt build cli, made region (now '-r') an optional argument. PR #278
- If the model root already contains files when setting root, this will cause an error unless force overwrite (mode='w+' or --fo/--force-overwrite from command line). PR #278
- Revamped the GeoDataset (vector.py) to now work with geometry objects and wkt strings besides xy coordinates. PR #276
- GeoDataset can write to .nc that is compliant with ogr. PR #208
- Support for rotated grids in RasterDataset/Array, with new rotation and origin properties. PR #272
- Removed pygeos as an optional dependency, hydromt now relies entirely on shapely 2.0 PR #258
- Changed shapely to require version '2.0.0' or later. PR #228
- strict and consistent read/write mode policy PR #238
- do not automatically read hydromt_data.yml file in model root. PR #238
- RasterDataset zarr driver: possiblility to read from several zarr stores. The datasets are then merged and
preprocess
can
be applied similar to netcdf driver. PR #249
Added
- New methods to compute PET in workflows.forcing.pet using Penman Monteith FAO-56 based on the
pyet
module. Available arguments are now method = ['debruin', 'makkink', 'penman-monteith_rh_simple', 'penman-monteith_tdew'] PR #266 - New get_region method in cli/api.py that returns a geojson representation of the parsed region. PR #209
- write raster (DataArray) to tiles in xyz stucture with the RasterDataArray.to_xyz_tiles method. PR #262
- add zoom_level to DataCatalog.get_rasterdataset method. PR #262
- new write_vrt function in gis_utils to write '.vrt' using GDAL. PR #262
- new predefined catalog for cmip6 data stored on Google Cloud Storage
cmip6_data
. Requires dependency gcsfs. PR #250 - new predefined catalog for public data stored on Amazon Web Services
aws_data
. Requires dependency s3fs. PR #250 - new DataCatalog preprocess function
harmonise_dims
for manipulation and harmonisation of array dimensions. PR #250 - experimental: support for remote data with a new yml data source
filesystem
attribute. Supported filesystems are [local, gcs, s3].
Profile information can be passed in the data catalogkwargs
under storage_options. PR #250 - experimental: new caching option for tiled rasterdatasets ('--cache' from command line). PR #286
Fixed
- bug related to opening named raster files. PR #262
- All CRS objects are from pyproj library (instead of rasterio.crs submodule). PR #230
- fix reading lists and none with config. PR #246
- fix
DataCatalog.to_yml
andDataCatalog.export()
with relative path and add meta section. PR #238
Deprecated
x_dim
,y_dim
, andtotal_bounds
attributes of GeoDataset/GeoDataArray are renamed tox_name
,y_name
andbounds
. PR #276- Move pygeos to optional dependencies in favor of shapely 2.0. PR #228
- Resolution option in hydromt build cli. PR #278
Documentation
- Added Working with GeoDatasets python notebook. PR #276
- added working_with_models example notebook. PR #229
- added export_data example notebook. PR #222
- added reading_point_data example notebook. PR #216
- added working_with_flow_directions example notebook. PR #231
- added prep_data_catalog example notebook. PR #232
- added reading_tabular_data example notebook. PR #216
Full list of PRs (auto generated):
- Era5 vars by @hboisgon in #187
- Prep 0.6 by @hboisgon in #223
- Post release commit by @hboisgon in #224
- Working with models example by @hboisgon in #229
- added export_data notebook by @hboisgon in #222
- Example point data by @IMarth in #216
- add flow direction example #220 by @DirkEilander in #231
- simplify binder using latest release by @DirkEilander in #233
- Prepare data catalog notebook by @hboisgon in #232
- Small notebook updates by @hboisgon in #234
- Example docs improvements by @JoostBuitink in #235
- Datacatalog fixes and strict read/write mode in Model class by @DirkEilander in #238
- update data_artifacts to v0.0.7 by @DirkEilander in #255
- fixed reading lists and none with config by @DirkEilander in #246
- Checklist for pull request added. Ref #264 by @couasnonanais in #265
- Api get region by @Tjalling-dejong in #209
- Envyml by @B-Dalmijn in #268
- fix eval setup_config by @DirkEilander in #275
- make pygeos an optional dependency by @DirkEilander in #258
- CLI revision (i.e. region & res) by @B-Dalmijn in #277
- notebook for tabular data by @IMarth in #217
- v0.0.8 artifact data by @laurenebouaziz in #280
- Penman by @DirkEilander in #266
- CLI revision (i.e. region & res); Reopend by @B-Dalmijn in #278
- update gpkg files to fgb in deltares_data catalog by @DirkEilander in #240
- support for rotated grid by @DirkEilander in #272
- Geodataset by @B-Dalmijn in #276
- Draft: support CMIP6 from Google Cloud by @hboisgon in #250
- Tiles: DataArray to xyz by @B-Dalmijn in #262
- fix name error in GeoDataArray.from_gdf by @DirkEilander in #288
- add vrt tile caching functionality by @DirkEilander in #286
New Contributors
- @B-Dalmijn made their first contribution in #268
Full Changelog: v0.6.0...v0.7.0
v0.6.0
v0.6.0 (24 October 2022)
In this release, we updated the Model API
by renaming staticgeoms to geoms, adding a new maps object and removing abstract methods. We also added new general subclasses to Model: GridModel
, LumpedModel
, MeshModel
, NetworkModel
. These new subclasses have their own objects (e.g. grid for GridModel representing regular grids which replaces the old staticmaps object). More details in the list below:
Added
- ModelCatalog to discover generic and plugin model classes.
PR #202 <https://github.com/Deltares/hydromt/pull/202>
_ - Support for 2-dimensional tabular data through the new DataFrameAdapter.
PR #153 <https://github.com/Deltares/hydromt/pull/153>
_ - API calls to get info about model components and dataset for the dashboard.
PR #118 <https://github.com/Deltares/hydromt/pull/118>
_ - New submodelclasses in hydromt:
GridModel
,LumpedModel
,MeshModel
,NetworkModel
- Added entrypoints for lumped_model, mesh_model, grid_model
- New mixin classes created for model specific object:
GridMixin
for self.grid,LumpedMixin
for self.response_units,MeshMixin
for self.mesh,
MapsMixin
for self.maps - New high-level object: self.maps for storing regular rasters data (which can have resolution and / or projection).
- Maps generic setup methods:
MapsMixin.setup_maps_from_raster
andMapsMixin.setup_maps_from_rastermapping
- Mesh generic setup methods:
MeshModel.setup_mesh
,MeshMixin.setup_maps_from_raster
andMeshMixin.setup_maps_from_rastermapping
Changed
- self.staticgeoms object and methods renamed to self.geoms
- self.staticmaps object and methods renamed to self.grid and moved into GridModel and GridMixin
Fixed
- Bug in backward compatibility of staticgeoms (not read automatically).
Issue #190 <https://github.com/Deltares/hydromt/issues/190>
_ - Direct import of xarray.core.resample.
Issue #189 <https://github.com/Deltares/hydromt/issues/189>
_ - Bug in dim0 attribute of raster, removed instead of set to None if no dim0
Issue #210 <https://github.com/Deltares/hydromt/issues/210>
Deprecated
- self.staticgeoms and self.staticmaps are deprecated.
v0.5.0
v0.5.0 (4 August 2022)
We now use rioxarray to read raster data. We recommend reinstalling your hydromt environment including the rioxarray package.
Furthermore, we have changed some internals to support multiple predefined data catalogs and data archives. A detailed overview of all changes is shown below.
@IMarth and @xldeltares made their first contributions 🎉
Added
- New raster method for adding gdal_compliant() attributes to xarray object.
- Function
to_datetimeindex
in available preprocess functions for xr.open_dataset in the data adapter. - Function
remove_duplicates
in available preprocess functions for xr.open_dataset in the data adapter. - New
DataCatalog.from_predefined_catalogs
andDataCatalog.from_archive
to support predefined data catalogs and archive
in a generic way through the data/predefined_catalogs.yml file. - Optional formatting for year and month variables in path of data sources.
Changed
- splitted data_adapter.py into a data_catalog and data_adapter submodule with py scripts per adapter
- Add rioxarray dependency to read raster data
- In build or update methods, the setup_config component is not forced to run first anymore but according to order of the components in the ini config (opt dict).
- In DataCatalog.get_RasterDataset & DataCatalog.get_GeoDataset methods, variables can now also be a str as well as a list of strings.
- In DataCatalog.get_RasterDataset & DataCatalog.get_GeoDataset methods, automatic renaming of single variable datasets based on the variables argument will be deprecated
- Interpolate missing values based on D4 neighbors of missing value cells only. This largely improves the performance without loosing accuracy.
Changes have been observed whennearest
method is used but this should not impact quality of the interpolation. - New source_names argument to DataCatalog.to_yml
Fixed
- Fixed DataAdapter.resolve_paths with unknown keys #121
- Fixed the WGS84 datum in the gis_utils.utm_crs method.
- In merge.merge the grid is now aligned with input dataset with the largest overlap if no dst_bounds & dst_res are given.
- Fixed the predicate not being passed in get_geodataframe method.
- Removed deprecated xr.ufuncs calls.
Deprecated
- Automatic renaming of single var dataset if variables is provided in get_rasterdataset. Data catalog should be used instead.
DataCatalog.from_artifacts
. UseDataCatalog.from_predefined_catalogs
instead.