Datacube v1.8.4

v1.8.4 (6 August 2021)

• Removed example and contributed notebooks from the repository. Better notebook examples exist
• Removed datacube_apps, as these are not used and not maintained
• Add cloud_cover to EO3 metadata
• Add erosion functionality to Virtual products' ApplyMask to supplement existing dilation functionality #1049
• Fix numeric precision issues in compute_reproject_roi when pixel size is small #1047
• Follow up fix to #1047 to round scale to nearest integer if very close
• Add support for 3D Datasets #1099
• New feature: search by URI from the command line datacube dataset uri-search
• Added new "license" and "description" properties to DatasetType to enable easier access to product information #1143 #1144
• Revised the Datacube.list_products function to produce a simpler and more useful product list table #1145
• Refactor docs, making them more up to date and simpler #1137 #1128
• Add new dataset_predicate param to dc.load and dc.find_datasets for more flexible temporal filtering (e.g. loading data for non-contiguous time ranges such as specific months or seasons over multiple years) #1148 #1156
• Fix to GroupBy to ensure output output axes are correctly labelled when sorting observations using sort_key #1157
• GroupBy is now its own class to allow easier custom grouping and sorting of data #1157
• add support for IAM authentication for RDS databases in AWS #1168

1.8.3

• More efficient band alias handling
• More documentation cleanups
• Bug fixes in datacube.utils.aws, credentials handling when AWS_UNSIGNED is set
• Product definition can now optionally include per-band scaling factors
• Fix issue where new updated columns aren't created on a fresh database
• Fix bug around adding updated columns locking up active databases

1.8.2

• Fix regressions in .geobox (#982)
• Expand list of supported dtypes to include complex values (#989)
• Can now specify dataset location directly in the yaml document (#990, #989)
• Better error reporting in datacube dataset update (#983)

1.8.1

Summary

This release contains mostly bug fixes an documentation improvements.

Full List of Changes

• Added an updated column for trigger based tracking of database row updates in PostgreSQL. (#951)
• Changes to the writer driver API. The driver is now responsible for constructing output URIs from user configuration. (#960)
• Added a datacube.utils.geometry.assign_crs method for better interoperability with other libraries (#967)
• Better interoperability with xarray - the xarray.Dataset.to_netcdf function should work again (#972, #976)
• Add support for unsigned access to public S3 resources from CLI apps (#976)
• Usability fixes for indexing EO3 datasets (#958)
• Fix CLI initialisation of the Dask Distributed Executor (#974)

1.8.0

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by dc.load have changed in a significant way (#837, #899).
2. We no longer use GDAL native Python bindings (osgeo.{ogr,osr}) and instead rely on pyproj and shapely as a backend for Geometry and CRS classes (#880).

We no longer store CRS as an object (datacube.utils.geometry.CRS) in an attribute dictionary of the DataArray, instead it is stored in a string format (WKT) in an attribute of a special spatial_ref coordinate. This change allows us to better interface with xarray IO libraries. One can now write data loaded by dc.load with xr.to_netcdf(..) directly and load back with xr.open_dataset(..), all while maintaining geo-registration information (i.e. .geobox property).

Also, since CRS information is now stored on the Coordinate, and not on the DataArray itself, it survives a greater variety of mathematical operations. Attributes on the DataArray would often go missing when doing the most basic of operations, like changing dtype of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of gdal python library. Both shapely and pyproj that replaced it, offer binary wheels, and are therefore much simpler to install.

Changes since 1.8.0rc1

• Expanded EO3 support
• Bug fixes in EO3 handling
• Cleanup in docs
• Better compatibility with other libraries for CRS construction
• Removed ancient db migration code

Full List of Changes

• Changed geo-registration mechanics for arrays returned by dc.load
• Migrate geometry and CRS backends from osgeo.ogr and osgeo.osr to shapely and pyproj respectively
• Fixes for geometries crossing anti meridian
• EO3 dataset metadata format is now understood by datacube dataset add
• New virtual product combinator reproject for on-the-fly reprojection of rasters
• Enhancements to the expressions transformation in virtual products
• Support /vsi** style paths for dataset locations
• Remove old Search Expressions and replace with a simpler implementation based on Lark Parser
• Remove no longer required PyPEG2 dependency
• Change development version numbers generation. Use setuptools_scm instead of versioneer
• Deprecated datacube.helpers.write_geotiff, use datacube.utils.cog.write_cog for similar functionality
• Deprecated datacube.storage.masking, moved to datacube.utils.masking
• Remove S3AIO driver
• Removed migration support from datacube releases before 1.1.5.
  If you still run a datacube before 1.1.5 (from 2016 or older), you will need to update it
  using ODC 1.7 first, before coming to 1.8.

1.8.0rc1

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by dc.load have changed in a significant way (#837, #899).
2. We no longer use GDAL native Python bindings (osgeo.{ogr,osr}) and instead rely on pyproj and shapely as a backend for Geometry and CRS classes (#880).

We no longer store CRS as an object (datacube.utils.geometry.CRS) in an attribute dictionary of the DataArray, instead it is stored in a string format (WKT) in an attribute of a special spatial_ref coordinate. This change allows us to better interface with xarray IO libraries. One can now write data loaded by dc.load with xr.to_netcdf(..) directly and load back with xr.open_dataset(..), all while maintaining geo-registration information (i.e. .geobox property).

Also, since CRS information is now stored on the Coordinate, and not on the DataArray itself, it survives a greater variety of mathematical operations. Attributes on the DataArray would often go missing when doing the most basic of operations, like changing dtype of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of gdal python library. Both shapely and pyproj that replaced it, offer binary wheels, and are therefore much simpler to install.

Full List of Changes

• Changed geo-registration mechanics for arrays returned by dc.load
• Migrate geometry and CRS backends from osgeo.ogr and osgeo.osr to shapely and pyproj respectively
• Fixes for geometries crossing anti meridian
• EO3 dataset metadata format is now understood by datacube dataset add
• New virtual product combinator reproject for on-the-fly reprojection of rasters
• Enhancements to the expressions transformation in virtual products
• Support /vsi** style paths for dataset locations
• Remove old Search Expressions and replace with a simpler implementation based on Lark Parser
• Remove no longer required PyPEG2 dependency
• Change development version numbers generation. Use setuptools_scm instead of versioneer
• Deprecated datacube.helpers.write_geotiff, use datacube.utils.cog.write_cog for similar functionality
• Deprecated datacube.storage.masking, moved to datacube.utils.masking
• Remove S3AIO driver

1.8.0b6

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by dc.load have changed in a significant way (#837, #899).
2. We no longer use GDAL native Python bindings (osgeo.{ogr,osr}) and instead rely on pyproj and shapely as a backend for Geometry and CRS classes (#880).

We no longer store CRS as an object (datacube.utils.geometry.CRS) in an attribute dictionary of the DataArray, instead it is stored in a string format (WKT) in an attribute of a special spatial_ref coordinate. This change allows us to better interface with xarray IO libraries. One can now write data loaded by dc.load with xr.to_netcdf(..) directly and load back with xr.open_dataset(..), all while maintaining geo-registration information (i.e. .geobox property).

Also, since CRS information is now stored on the Coordinate, and not on the DataArray itself, it survives a greater variety of mathematical operations. Attributes on the DataArray would often go missing when doing the most basic of operations, like changing dtype of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of gdal python library. Both shapely and pyproj that replaced it, offer binary wheels, and are therefore much simpler to install.

Full List of Changes

• Changed geo-registration mechanics for arrays returned by dc.load
• Migrate geometry and CRS backends from osgeo.ogr and osgeo.osr to shapely and pyproj respectively
• New virtual product combinator reproject for on-the-fly reprojection of rasters
• Enhancements to the expressions transformation in virtual products
• Support /vsi** style paths for dataset locations
• Remove old Search Expressions and replace with a simpler implementation based on Lark Parser
• Remove no longer required PyPEG2 dependency
• Change development version numbers generation. Use setuptools_scm instead of versioneer
• Deprecated datacube.helpers.write_geotiff, use datacube.utils.cog.write_cog for similar functionality
• Deprecated datacube.storage.masking, moved to datacube.utils.masking
• Remove S3AIO driver

1.8.0b5

Testing release process, please ignore this one.

1.7 (16 May 2019)

Not a lot of changes since rc1.

• Early exit from dc.load on KeyboardInterrupt, allows partial loads inside notebook.
• Some bug fixes in geometry related code
• Some cleanups in tests
• Pre-commit hooks configuration for easier testing
• Re-enable multi-threaded reads for s3aio driver (set use_threads=True in dc.load(..))

1.7rc1 (18 April 2019)

1.7rc1 (18 April 2019)

Virtual Products

Add Virtual Products for multi-product loading.

(#522, #597, #601, #612, #644, #677, #699, #700)

Changes to Data Loading

The internal machinery used when loading and reprojecting data, has been
completely rewritten. The new code has been tested, but this is a
complicated and fundamental part of code and there is potential for
breakage.

When loading reprojected data, the new code will produce slightly
different results. We don't believe that it is any less accurate than
the old code, but you cannot expect exactly the same numeric results.

Non-reprojected loads should be identical.

This change has been made for two reasons:

1. The reprojection is now core Data Cube, and is not the
   responsibility of the IO driver.
2. When loading lower resolution data, DataCube can now take advantage
   of available overviews.

• New futures based IO driver interface (#686)

Other Changes

• Allow specifying different resampling methods for different data
  variables of the same Product. (#551)
• Allow all reampling methods supported by rasterio. (#622)
• Bug fix (Index out of bounds causing ingestion failures)
• Support indexing data directly from HTTP/HTTPS/S3 URLs (#607)
• Renamed the command line tool datacube metadata_type to datacube metadata (#692)
• More useful output from the command line datacube {product|metadata} {show|list}
• Add optional progress_cbk to dc.load(_data) (#702), allows user
  to monitor data loading progress.
• Thread-safe netCDF access within dc.load (#705)

Performance Improvements

• Use single pass over datasets when computing bounds (#660)
• Bugfixes and improved performance of dask-backed arrays (#547,
  #664)

Documentation Improvements

• Improve API Reference documentation.

Deprecations

• From the command line, the old query syntax for searching within
  vague time ranges, eg: 2018-03 < time < 2018-04 has been removed.
  It is unclear exactly what that syntax should mean, whether to
  include or exclude the months specified. It is replaced by time in [2018-01, 2018-02] which has the same semantics as dc.load time
  queries. (#709)