Discussion on intake-esm's roadmap and potential better integration with ESGF

[andersy005]

The following is copied from an email I received from @stephank16 from German Climate Computing Center (DKRZ). @stephank16 and I agreed that it would be better to move this discussion here so that we can engage other contributors/collaborators as well.

---

Some background:

Core ESGF developers met late last year in a "ESGF future architecture"
workshop and as part of this we also shortly discussed potential roles
of intake-esm (and maybe dedicated intake-esm/ESGF extensions,
adaptations) .. (I saw that the ESGF search interface is mentioned as a
"related work" to intake-esm in the intake-esm documentation.. .)

Just as a concrete example one possibility of such a ESGF/intake-esm
integration could be to use a (solr or elastic-search) index as a
replacement of the current file based catalog backend to better support
larger user groups using the intake-esm search options on larger local
CMIP data pools etc. ( e.g. there is an intake-elasticsearch extension,
so maybe having also an intake-esm-elasticsearch extension ... ? ) - but
this is just one option ..

So I would have have some questions:

• do you think such a discussion makes sense ? whom to include from
  pangeo/NCAR/.. side ? (we are currently thinking about a very small
  group of people from ESGF side - probably one from LLNL, one from DKRZ
  (Germany), one from France (IPSL) and one from the UK (STFC) )

• what is the roadmap for intake-esm (e.g. are you also thinking about
  such a search backend, is the catalog spec stable etc ..- STAC .. ? )

Thanks a lot and hoping to get a discussion started ..

[andersy005]

Soliciting input from @kmpaul, @matt-long, @rabernat, @jhamman, @naomi-henderson

[jhamman]

Here's what we've done with STAC and satsearch (the Python client for dynamic searching of STAC APIs -- usually elasticsearch on the server side).

• Provide a simple Intake-stac interface that allows for loading of STAC catalogs into Intake
• Provide hooks for satsearch results to ingest directly into intake-stac

In practice, this plays out like this:

import satsearch
from intake import open_stac_item_collection as open_stac

# use sat search to query a dynamic STAC API
properties = ["eo:row=027", "eo:column=047", "landsat:tier=T1"] 
results = satsearch.Search.search(
    collection='landsat-8-l1', 
    sort=['<datetime'], #earliest scene first
    property=properties)

sceneid = 'LC80470272019096'
catalog = open_stac(results.items())  # convert the satsearch results to intake
da = catalog[sceneid]['B2'].to_dask()  # Return an xarray dataset backed by dask

One idea would be to work towards something like this for use with ESGFs search APIs and intake-esm. I personally don't think STAC is quite ready for global model datasets with long time spans so I'm not sure it would be worth investing too much time in that option.

[kmpaul]

Sorry it's been a long time since this was brought up.

I'm not aware of any long-term plans with intake-esm that these plans would collide with. However, it seems to me that maybe starting with a prototype intake-esgf plugin would be the best approach? If there is clear overlap after that gets build, then it seems like we can talk about integration. However, it's not clear to me that there is an enormous amount of overlap between what intake-esgf would need and what intake-esm provides.

If I'm wrong, maybe someone could clarify that for me.

[andersy005]

It appears that the ESGF funding is going away, and there is ongoing effort to redevelop the new ESGF 2.0.... I’m closing this issue for the time being. Please reopen if you find this issue relevant in the short-term.

[andersy005]

It turns out that there are relevant discussions about inkake/intake-esm/STAC in the new ESGF architecture. So, I am reopening this for future reference....