Replacement for from_df in version 2022.9.18 and later?

[wrongkindofdoctor]

I'm updating the esm-intake package and dependencies in the conda environments for MDTF-diagnostics in anticipation of rewriting the preprocessor to use esm intake catalogs for data querying. The preprocessor in the latest package version uses the from_df method in the core.esm_datastore in intake-esm v2021.8.17 to generate a catalog from Pandas dataframe contents:

 def setup_query(self):
        """Generate an intake_esm catalog of files found in CATALOG_DIR.
        Attributes of files listed in the catalog (columns of the DataFrame) are
        taken from the match groups (fields) of the class's \_FileRegexClass.
        """
        self.log.info('Starting data file search at %s:', self.CATALOG_DIR)
        # df: DataFrame, catalog content that would otherwise be in a csv file
        # esmcol_data: Optional[Dict[str, Any]] = None, ESM collection spec information
        # progressbar: bool = True, if True, prints progress bar to standard error when loading info into xarray dataset
        # sep: str = '.', delimiter to use when constructing key for a query
        # **kwargs: Any

        self.catalog = intake_esm.core.esm_datastore.from_df(
            self.generate_catalog(),
            esmcol_data=self._dummy_esmcol_spec(),
            progressbar=False, sep='|'
        )

from_df is deprecated in the latest release, and it is not clear what the replacement method(s) should be.

self.catalog = intake_esm.core.esm_datastore.from_df(
AttributeError: type object 'esm_datastore' has no attribute 'from_df'

If someone could point me in the right direction, I'd appreciate it.

[andersy005]

@wrongkindofdoctor, apologies for the inconveniences that were caused by the breaking changes. In hindsight, we should have kept the from_dict() method but warned that it would be removed in the newest version.

when you get a moment, try the following:

obj = {'df' : self.generate_catalog(), 'esmcat':  self.dummy_esmcol_spec()}

self.catalog = intake_esm.core.esm_datastore(
            obj,
            progressbar=False, sep='|'
        )

[wrongkindofdoctor]

@andersy005 No problem. I subbed the suggested code snippet, but it triggered the following error in cat.py:

 File "/home/jessica.liptak/mdtf/MDTF-diagnostics/src/query_fetch_preprocess.py", line 850, in request_data
    self.pre_query_and_fetch_hook()
  File "/home/jessica.liptak/mdtf/MDTF-diagnostics/src/query_fetch_preprocess.py", line 100, in pre_query_and_fetch_hook
    self.setup_query()
  File "/home/jessica.liptak/mdtf/MDTF-diagnostics/src/query_fetch_preprocess.py", line 198, in setup_query
    self.catalog = intake_esm.core.esm_datastore(
  File "/home/jessica.liptak/miniconda3/envs/_MDTF_base/lib/python3.10/site-packages/intake_esm/core.py", line 92, in __init__
    self.esmcat = ESMCatalogModel.from_dict(obj)
  File "/home/jessica.liptak/miniconda3/envs/_MDTF_base/lib/python3.10/site-packages/intake_esm/cat.py", line 143, in from_dict
    cat = cls.parse_obj(esmcat)
  File "pydantic/main.py", line 527, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ESMCatalogModel
aggregation_control
  field required (type=value_error.missing)

The esmcat object passed to cls.parse_obj(esmcat) looks like this:

{'esmcat_version': '0.1.0', 'id': 
'MDTF_MultirunLocalFileDataSource', 
'description': ' ',
 'attributes': 
[{'column_name': 'sample_dataset', 'vocabulary': ''},
 {'column_name': 'frequency', 'vocabulary': ''}, 
{'column_name': 'variable', 'vocabulary': ''}],
 'assets': {'column_name': 'remote_path', 'format': 'netcdf'},
 'last_updated': '2020-12-06'}

Diving into the underlying Python 3.10 code places me in the enum.__call__ method:

   def __call__(cls, value, names=None, *, module=None, qualname=None, type=None, start=1):
       """
       Either returns an existing member, or creates a new enum class.

       This method is used both when an enum class is given a value to match
       to an enumeration member (i.e. Color(3)) and for the functional API
       (i.e. Color = Enum('Color', names='RED GREEN BLUE')).

       When used for the functional API:

       `value` will be the name of the new class.

       `names` should be either a string of white-space/comma delimited names
       (values will start at `start`), or an iterator/mapping of name, value pairs.

       `module` should be set to the module this class is being created in;
       if it is not set, an attempt to find that module will be made, but if
       it fails the class will not be picklable.

       `qualname` should be set to the actual location this class can be found
       at in its module; by default it is set to the global scope.  If this is
       not correct, unpickling will fail in some circumstances.

       `type`, if set, will be mixed in as the first base class.
       """
       if names is None:  # simple value lookup
           return cls.__new__(cls, value)
       # otherwise, functional API: we're creating a new Enum type
       return cls._create_(
               value,
               names,
               module=module,
               qualname=qualname,
               type=type,
               start=start,
               )

Where the code fails in

  if names is None:  # simple value lookup
            return cls.__new__(cls, value)

for value='netcdf' (which is a str)
The last step (bear with me) in the call stack ends in the enum.__new__ method that attempts to return the result of cls._value2member_map_[value], which is DataFormat.netcdf:

    def __new__(cls, value):
        # all enum instances are actually created during class construction
        # without calling this method; this method is called by the metaclass'
        # __call__ (i.e. Color(3) ), and by pickle
        if type(value) is cls:
            # For lookups like Color(Color.RED)
            return value
        # by-value search for a matching enum member
        # see if it's in the reverse mapping (for hashable values)
        try:
            return cls._value2member_map_[value]
        except KeyError:
            # Not found, no need to do long O(n) search
            pass

I apologize for poking a bug in the the new version if that is the case. LMK if I can provide additional information, or if there's something wrong with the contents in the esmcat generated by MDTF.

[andersy005]

we're making progress :) it sounds like you are running into the issue that this PR is trying to fix:

• Optional aggregation control #569

i'm curious as to what happens when you add the 'aggregation_control': { 'variable_column_name': 'variable', 'groupby_attrs': [] } to your esmcat dictionary

{'esmcat_version': '0.1.0', 'id': 
'MDTF_MultirunLocalFileDataSource', 
'description': ' ',
 'attributes': 
[{'column_name': 'sample_dataset', 'vocabulary': ''},
 {'column_name': 'frequency', 'vocabulary': ''}, 
{'column_name': 'variable', 'vocabulary': ''}],
 'assets': {'column_name': 'remote_path', 'format': 'netcdf'},
 'last_updated': '2020-12-06',
'aggregation_control': {
'variable_column_name': 'variable', 'groupby_attrs': []
}
}

[wrongkindofdoctor]

@andersy005 Thanks! This seems to have resolved the esmcat issue, but I've tripped something else in cf_xarray downstream (ah, dependencies), so I have some more debugging to do on my end.

[andersy005]

good luck with debugging the cf-xarray issue :) please let us know if you encounter any other issues