Datacube 1.9 and 2.0 discussion

[SpacemanPaul]

Welcome Open Data Cube users.

The ODC Steering Council and developers have been working hard behind the scenes over the last year or so, addressing the worst of the accumulated technical debt in datacube-core and preparing for new major releases.

The plan includes several backwards-incompatible changes. An effort has been made to provide a smooth migration pathway wherever possible, with existing behaviour in 1.8.x versions being deprecated in 1.9.x, with alternatives being provided, then removing the deprecated behaviour in 2.0.x with the alternative approach becoming the standard. There will be some minor backwards incompatible changes in 1.9.0, but these should mostly affect administrators and developers - the top level APIs that most sandbox users use should not see any breaking changes.

We do not yet have an expected release date for 1.9.0 but we are hopeful that it will happen in the first few months of 2024.

After the release of 1.9.0, focus will shift to updating secondary ODC libraries to work with ODC-1.9. (Explorer and OWS in particular will need major updates). We will continue to support and release 1.8.x versions after the release of 1.9.0, until the 1.9.x releases have stabilised and all secondary libraries are up to date.

Here's a brief summary of what to expect from datacube-1.9:

• The old datacube.utils.geometry library will be replaced by odc-geo . If you have used odc-geo you will understand how much additional power and flexibility this brings to core.
• The index driver API will be cleaned up and simplified, facilitating easier development of new index backends in future.
  A new PostGIS-based index backend. (The legacy Postgres index driver will continue to be supported in 1.9, but will be dropped in ODC-2.0.)
• The Postgis index driver will only support EO3-compatible metadata types. Older EO-style metadata types should be migrated to EO3. We will try to provide tools to assist with this migration, but due to the arbitrary generality of pre-EO3 ODC metadata, such tools may not be possible in all cases. (The legacy postgres driver will continue to support non-EO3 metadata types until it is dropped in 2.0)
• The postgis driver will support the creation of PostGIS spatial indexes for arbitrary CRSs. This will improve efficiency and accuracy of database searches, particularly when working with data covering regions where EPSG:4326 lat/long coordinates are non-linear (e.g. the Pacific around the anti-meridian and particularly the north and south polar regions).
• A new configuration engine will replace the existing system. Most existing configuration files should continue to work as previously, but there are some backwards-incompatible changes - in particular from 1.9.0 only a single configuration file will be read (previously multiple configuration files could be merged on top of each other), and the way in which a default config environment is specified is slightly different.
• The postgis driver handles lineage very differently to the postgres driver: Lineage data is only loosely coupled to dataset metadata and a completely new API is introduced for working with lineages. It is now possible to store external lineage information - i.e. it is not necessary for both the source and derived dataset to exist in the index for the lineage relationship between them to be recorded in the database.
• Support for multi-dimensional loading of hyperspectral datasets

More details can be found by reading the ODCv2 roadmap and the various Enhancement Proposals - all of the most significant changes are discussed there in more detail.

Feedback from the ODC community is encouraged - we want to make the transition as smooth as possible for all users and ensure that nobody gets left behind. You can leave comments here - please include which Enhancement Proposal your comment is in relation to where relevant.

[alexgleith]

store external lineage information

What does this look like? Is it a list of URLs?

[SpacemanPaul]

A list of UUIDs - or more accurately a list of source-derived UUID pairs.

There is also a second table that can optionally label external UUIDs with the "home" database in which they can be found.

The new API can work with arbitrarily deep tree structures containing lineage and home database information, and follow relationships in either direction (sourcewards or derivedwards).

The new API is described in detail in EP-08.

[robbibt]

This is very exciting, thanks for the excellent documentation @SpacemanPaul!

Regarding odc-geo, how will importing geometry-related functions/tools work in the future (e.g. things like datacube.utils.geometry.Geometry)? Will those functions still be available via the same datacube module namespace (not sure if that's the right term) we use now (but loading odc-geo code under the hood), or will users need to import odc-geo manually to access them?

(it would be nice if users didn't have to load an additional package to access functionality that used to be available via simply a datacube import)

[Kirill888]

Align is more tricky. Specifying align in CRS units was a mistake. It's painful to work with that, especially when data is in geographic coordinates with high resolution. Also name is very confusing, in my opinion.

In odc-geo this mechanism is called "anchor", sample usage https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geobox.GeoBox.from_bbox.html

And you can specify either EDGE or CENTER via this enum:

https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geobox.AnchorEnum.html#odc.geo.geobox.AnchorEnum

Or you can specify arbitrary "alignment" with an XY[float] tuple in fraction of a pixel units:

• EDGE == xy_(0, 0)
• CENTER == xy_(0.5, 0.5)

It is basically location of the CRS coordinate (0.0, 0.0) within a pixel that would contain that location when pixel grid is expanded far enough.

[robbibt]

Ah, neat! So for Landsat centre-aligned pixels we could use anchor="CENTER"? That would be much simpler and easier to explain to users.

In that case, should we add support for anchor in datacube-core in 1.9, and a deprecation warning for align?

(I think align is supported by "load hints", so we'd need to add support for anchor there too.)

[Kirill888]

currently text literal is not supported as input, but it should be and it's easy enough to add.

[Ariana-B]

@Kirill888 It would be great if more details about migrating from align to anchor were included in the migration guide.

[SpacemanPaul]

Resolution already prefers a single int in 1.9. (Supplying a tuple assumes yx order and warns to use odc.geo.resxy_ or resyx)

Agree we should deprecate align in favour of anchor (and propagate anchor out to all the places it needs to be).