Error when accessing MRI-ESM2-0 #561

axelschweiger · 2023-01-23T02:32:33Z

axelschweiger
Jan 23, 2023

Here's a quick checklist in what to include:

Include a detailed description of the bug or suggestion
Output of intake_esm.show_versions()
Minimal, self-contained copy-pastable example that generates the issue if possible. Please be concise with code posted. See guidelines below on how to provide a good bug report:
- Minimal Complete Verifiable Examples
- Craft Minimal Bug Reports
  
  Bug reports that follow these guidelines are easier to diagnose,
  and so are often handled much more quickly.

Description

I'm trying to access some CMIP6 data using intake-esm with below code.
Running this generates the following error. I'm new to intake so I have no clue what's going on here. Something about mixed time objections. Any pointers greatly appreciated.

Traceback

The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
/home/axel/ax_python_lib/CMIP6/intake_debug.py:24: DeprecationWarning: cdf_kwargs and zarr_kwargs are deprecated and will be removed in a future version. Please use xarray_open_kwargs instead.
  dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True})
Traceback (most recent call last):
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/source.py:258 in _open_dataset
    self._ds = xr.combine_by_coords(
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/xarray/core/combine.py:982 in combine_by_coords
    concatenated = _combine_single_variable_hypercube(
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/xarray/core/combine.py:629 in _combine_single_variable_hypercube
    combined_ids, concat_dims = _infer_concat_order_from_coords(list(datasets))
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/xarray/core/combine.py:131 in _infer_concat_order_from_coords
    _ensure_same_types(series, dim)
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/xarray/core/combine.py:78 in _ensure_same_types
    raise TypeError(error_msg)
TypeError: Cannot combine along dimension 'time' with mixed types. Found: DatetimeProlepticGregorian, Timestamp. If importing data directly from a file then setting `use_cftime=True` may fix this issue.
Traceback (most recent call last):
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)
  File ~/ax_python_lib/CMIP6/intake_debug.py:24
    dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True})
  File pydantic/decorator.py:40 in pydantic.decorator.validate_arguments.validate.wrapper_function
  File pydantic/decorator.py:134 in pydantic.decorator.ValidatedFunction.call
  File pydantic/decorator.py:206 in pydantic.decorator.ValidatedFunction.execute
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/core.py:651 in to_dataset_dict
    raise exc
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/core.py:647 in to_dataset_dict
    key, ds = task.result()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/concurrent/futures/_base.py:451 in result
    return self.__get_result()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/concurrent/futures/_base.py:403 in __get_result
    raise self._exception
  File ~/anaconda3/envs/pangeo310/lib/python3.10/concurrent/futures/thread.py:58 in run
    result = self.fn(*self.args, **self.kwargs)
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/core.py:789 in _load_source
    return key, source.to_dask()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/source.py:273 in to_dask
    self._load_metadata()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake/source/base.py:285 in _load_metadata
    self._schema = self._get_schema()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/source.py:205 in _get_schema
    self._open_dataset()
  File ~/anaconda3/envs/pangeo310/lib/python3.10/site-packages/intake_esm/source.py:265 in _open_dataset
    raise ESMDataSourceError(
ESMDataSourceError: Failed to load dataset with key='ScenarioMIP.MRI.MRI-ESM2-0.ssp585.SImon.gn'
                 You can use `cat['ScenarioMIP.MRI.MRI-ESM2-0.ssp585.SImon.gn'].df` to inspect the assets/files for this key.

What I Did

rom matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr
import intake
import fsspec
import dask

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(cat_url)

cat = col.search(experiment_id=['historical', 'ssp585'], table_id='SImon', variable_id='sisnthick',
                 grid_label='gn',source_id='MRI-ESM2-0')
#                 grid_label='gn',source_id='CNRM-CM6-1-HR')
dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True})

Version information: output of `intake_esm.show_versions()`

INSTALLED VERSIONS

cftime: 1.6.2
dask: 2022.12.1
fastprogress: 1.0.3
fsspec: 2022.11.0
gcsfs: 2022.11.0
intake: 0.6.6
intake_esm: 2022.9.18
netCDF4: 1.6.2
pandas: 1.5.2
requests: 2.28.1
s3fs: 2022.11.0
xarray: 2022.12.0
zarr: 2.13.3

Answered by andersy005

Jan 23, 2023

@axelschweiger, thank you for filing this report. It appears the issue you are running had to do with mixed datetime types. One solution is to tell xarray to use cftime when decoding the time coordinates.

In [6]: dset_dict = cat.to_dataset_dict(xarray_open_kwargs={'use_cftime': True})

--> The keys in the returned dictionary of datasets are constructed as follows:
        'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
 |████████████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2/2 00:17<00:00]

In [7]: dset_dict
Out[7]: 
{'ScenarioMIP.MRI.MRI-ESM2-0.ssp585.SImon.gn': <xarray.Dataset>
 Dimensions:             (time…

View full answer

andersy005 · 2023-01-23T04:37:38Z

andersy005
Jan 23, 2023
Maintainer

@axelschweiger, thank you for filing this report. It appears the issue you are running had to do with mixed datetime types. One solution is to tell xarray to use cftime when decoding the time coordinates.

In [6]: dset_dict = cat.to_dataset_dict(xarray_open_kwargs={'use_cftime': True})

--> The keys in the returned dictionary of datasets are constructed as follows:
        'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
 |████████████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2/2 00:17<00:00]

In [7]: dset_dict
Out[7]: 
{'ScenarioMIP.MRI.MRI-ESM2-0.ssp585.SImon.gn': <xarray.Dataset>
 Dimensions:             (time: 3432, x: 360, y: 363, dcpp_init_year: 1,
                          member_id: 2, bnds: 2, vertices: 4)
 Coordinates:
   * time                (time) object 2015-01-16 12:00:00 ... 2300-12-16 12:0...
   * x                   (x) float64 0.0 1.0 2.0 3.0 ... 356.0 357.0 358.0 359.0
   * y                   (y) float64 -78.0 -77.5 -77.0 ... 149.7 151.8 154.0
   * dcpp_init_year      (dcpp_init_year) float64 nan
   * member_id           (member_id) object 'r1i1p1f1' 'r1i2p1f1'
     latitude            (y, x) float64 dask.array<chunksize=(363, 360), meta=np.ndarray>
     longitude           (y, x) float64 dask.array<chunksize=(363, 360), meta=np.ndarray>
     time_bnds           (time, bnds) object dask.array<chunksize=(3432, 2), meta=np.ndarray>
     vertices_latitude   (y, x, vertices) float64 dask.array<chunksize=(363, 360, 4), meta=np.ndarray>
     vertices_longitude  (y, x, vertices) float64 dask.array<chunksize=(363, 360, 4), meta=np.ndarray>
     x_bnds              (x, bnds) float64 dask.array<chunksize=(360, 2), meta=np.ndarray>
     y_bnds              (y, bnds) float64 dask.array<chunksize=(363, 2), meta=np.ndarray>
 Dimensions without coordinates: bnds, vertices
 Data variables:
     sisnthick           (member_id, dcpp_init_year, time, y, x) float32 dask.array<chunksize=(1, 1, 172, 363, 360), meta=np.ndarray>
 Attributes: (12/49)
     Conventions:                      CF-1.7 CMIP-6.2
     activity_id:                      ScenarioMIP
     branch_method:                    standard
     branch_time_in_child:             60265.0
     branch_time_in_parent:            60265.0
     cmor_version:                     3.5.0
     ...                               ...
     intake_esm_attrs:table_id:        SImon
     intake_esm_attrs:variable_id:     sisnthick
     intake_esm_attrs:grid_label:      gn
     intake_esm_attrs:version:         20210329
     intake_esm_attrs:_data_format_:   zarr
     intake_esm_dataset_key:           ScenarioMIP.MRI.MRI-ESM2-0.ssp585.SImon.gn,
 'CMIP.MRI.MRI-ESM2-0.historical.SImon.gn': <xarray.Dataset>
 Dimensions:             (y: 363, x: 360, member_id: 6, dcpp_init_year: 1,
                          time: 1980, bnds: 2, vertices: 4)
 Coordinates:
     latitude            (y, x) float64 dask.array<chunksize=(363, 360), meta=np.ndarray>
     longitude           (y, x) float64 dask.array<chunksize=(363, 360), meta=np.ndarray>
   * time                (time) object 1850-01-16 12:00:00 ... 2014-12-16 12:0...
     time_bnds           (time, bnds) object dask.array<chunksize=(1980, 2), meta=np.ndarray>
     vertices_latitude   (y, x, vertices) float64 dask.array<chunksize=(363, 360, 4), meta=np.ndarray>
     vertices_longitude  (y, x, vertices) float64 dask.array<chunksize=(363, 360, 4), meta=np.ndarray>
   * x                   (x) float64 0.0 1.0 2.0 3.0 ... 356.0 357.0 358.0 359.0
     x_bnds              (x, bnds) float64 dask.array<chunksize=(360, 2), meta=np.ndarray>
   * y                   (y) float64 -78.0 -77.5 -77.0 ... 149.7 151.8 154.0
     y_bnds              (y, bnds) float64 dask.array<chunksize=(363, 2), meta=np.ndarray>
   * member_id           (member_id) object 'r1i1p1f1' 'r1i2p1f1' ... 'r5i1p1f1'
   * dcpp_init_year      (dcpp_init_year) float64 nan
 Dimensions without coordinates: bnds, vertices
 Data variables:
     sisnthick           (member_id, dcpp_init_year, time, y, x) float32 dask.array<chunksize=(1, 1, 730, 363, 360), meta=np.ndarray>
 Attributes: (12/47)
     Conventions:                      CF-1.7 CMIP-6.2
     activity_id:                      CMIP
     branch_method:                    standard
     branch_time_in_child:             0.0
     cmor_version:                     3.5.0
     data_specs_version:               01.00.31
     ...                               ...
     intake_esm_attrs:table_id:        SImon
     intake_esm_attrs:variable_id:     sisnthick
     intake_esm_attrs:grid_label:      gn
     intake_esm_attrs:version:         20210311
     intake_esm_attrs:_data_format_:   zarr
     intake_esm_dataset_key:           CMIP.MRI.MRI-ESM2-0.historical.SImon.gn}

1 reply

axelschweiger Jan 23, 2023
Author

Thanks so much for the quick response. That did the trick!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when accessing MRI-ESM2-0 #561

{{title}}

{{editor}}'s edit

{{editor}}'s edit

INSTALLED VERSIONS

Replies: 1 comment 1 reply

{{title}}

{{title}}

Select a reply

Error when accessing MRI-ESM2-0 #561

axelschweiger Jan 23, 2023

Description

Traceback

What I Did

Version information: output of intake_esm.show_versions()

INSTALLED VERSIONS

Replies: 1 comment · 1 reply

andersy005 Jan 23, 2023 Maintainer

axelschweiger Jan 23, 2023 Author

axelschweiger
Jan 23, 2023

Version information: output of `intake_esm.show_versions()`

Replies: 1 comment 1 reply

andersy005
Jan 23, 2023
Maintainer

axelschweiger Jan 23, 2023
Author