[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

tomvothecoder · 2024-03-19T18:26:31Z

What happened?

There is an issue where transposing the shape of data causes misalignment with the original dimension order. When we attempt to preserve bounds, the original dimension order is still being used which does not align with the transposed dimension order.

Example

Let's say we have ds_a with an original dimension of shape of (lat: 1, lev: 2, lon: 3).

After transposing the data, output_data become shape=(2, 1, 3).

xcdat/xcdat/regridder/regrid2.py

Lines 164 to 172 in d0e33aa

    
           output_data_shape = [y_length, x_length] + other_sizes 
        
           output_data = output_data.reshape(output_data_shape) 
        
           output_order = [x + 2 for x in range(input_data_var.ndim - 2)] + [0, 1] 
        
           output_data = output_data.transpose(output_order) 
        
           return output_data.astype(np.float32)

We recreate the xr.DataArray (output_da) using output_data and dims. As mentioned above, the shape of output_data is now transposed, but it does not align with the original dims. This results in output_da having a new dimension shape of (lat: 2, lev: 1, lon: 3)

xcdat/xcdat/regridder/regrid2.py

Lines 184 to 197 in d0e33aa

    
           input_data_var = ds[data_var] 
        
           output_coords: dict[str, xr.DataArray] = {} 
        
           output_data_vars: dict[str, xr.DataArray] = {} 
        
           dims = list(input_data_var.dims) 
        
           output_da = xr.DataArray( 
        
               output_data, 
        
               dims=dims, 
        
               coords=output_coords, 
        
               attrs=ds[data_var].attrs.copy(), 
        
               name=data_var, 
        
           )

When we attempt to preserve lat_bnds in _preserve_bounds(), the original lat_bnds still has has a shape of (lat: 1, lev: 2, lon: 3), which does not align with (lat: 2, lev: 1, lon: 3).

xcdat/xcdat/regridder/base.py

Lines 48 to 58 in d0e33aa

    
           for ds in (output_grid, input_ds): 
        
               for axis in ("X", "Y", "Z", "T"): 
        
                   try: 
        
                       bnds = ds.bounds.get_bounds(axis) 
        
                   except KeyError: 
        
                       pass 
        
                   else: 
        
                       if bnds.name not in output_ds: 
        
                           output_ds[bnds.name] = bnds.copy() 
        
           return output_ds

This results in ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1).

What did you expect to happen? Are there are possible answers you came across?

Maybe output_data shape should align with the original dim order or we should also transpose the shape of the bounds?

Minimal Complete Verifiable Example (MVCE)

import numpy as np
import xarray as xr
import xcdat as xc

# 1. Create two datasets with dimensions (lat=1, lev=2, lon=3)
ds_a = xr.Dataset(
    {
        "lat": xr.DataArray(
            name="lat", dims="lat", data=[0], attrs={"axis": "Y", "bounds": "lat_bnds"}
        ),
        "lev": xr.DataArray(name="lev", dims="lev", data=[0, 1], attrs={"axis": "Z"}),
        "lon": xr.DataArray(
            name="lon", dims="lon", data=[0, 1, 2], attrs={"axis": "X"}
        ),
    }
)
ds_a["T"] = xr.DataArray(name="T", dims=["lat", "lev", "lon"], data=np.zeros((1, 2, 3)))
ds_a = ds_a.bounds.add_missing_bounds(axes=["X", "Y", "Z"])

# Manually add lat_bnds because xCDAT does not support adding bounds for
# singleton coords (yet)
ds_a["lat_bnds"] = xr.DataArray(name="lat_bnds", dims=["lat", "bnds"], data=[[-1, 1]])


ds_b = ds_a.copy()

# 2. Regrid dataset b to a using regrid2, bilinear
output_grid = ds_a.regridder.grid
ds_b_regrid = ds_b.regridder.horizontal(
    "T", output_grid, tool="regrid2", method="bilinear"
)
# ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1)

# 3. Breaks because `_regrid` incorrectly reorders dimensions
# Resulting in the `output_data` shape to become (lat=2, lev=1, lon=3).
˘

Relevant log output

ValueError                                Traceback (most recent call last)
File /global/u2/v/vo13/E3SM-Project/e3sm_diags/auxiliary_tools/cdat_regression_testing/657-meridional-mean-2d/657_meridional_mean_2d_run_script.py:19
     16 ds_b = ds_a.copy()
     18 output_grid = ds_a.regridder.grid
---> 19 ds_b_regrid = ds_b.regridder.horizontal(
     20     "T", output_grid, tool="regrid2", method="bilinear"
     21 )

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/accessor.py:324, in RegridderAccessor.horizontal(self, data_var, output_grid, tool, **options)
    322 input_grid = _get_input_grid(self._ds, data_var, ["X", "Y"])
    323 regridder = regrid_tool(input_grid, output_grid, **options)
--> 324 output_ds = regridder.horizontal(data_var, self._ds)
    326 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/regrid2.py:85, in Regrid2Regridder.horizontal(self, data_var, ds)
     79     input_data_var = input_data_var.where(src_mask != 0.0, masked_value)
     81 output_data = _regrid(
     82     input_data_var, src_lat_bnds, src_lon_bnds, dst_lat_bnds, dst_lon_bnds
     83 )
---> 85 output_ds = _build_dataset(
     86     ds,
     87     data_var,
     88     output_data,
     89     dst_lat_bnds,
     90     dst_lon_bnds,
     91     self._input_grid,
     92     self._output_grid,
     93 )
     95 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/regrid2.py:206, in _build_dataset(ds, data_var, output_data, dst_lat_bnds, dst_lon_bnds, input_grid, output_grid)
    199 output_data_vars[data_var] = output_da
    201 output_ds = xr.Dataset(
    202     output_data_vars,
    203     attrs=input_grid.attrs.copy(),
    204 )
--> 206 output_ds = _preserve_bounds(ds, output_grid, output_ds, ["X", "Y"])
    208 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/base.py:56, in _preserve_bounds(input_ds, output_grid, output_ds, drop_axis)
     54         else:
     55             if bnds.name not in output_ds:
---> 56                 output_ds[bnds.name] = bnds.copy()
     58 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/dataset.py:1603, in Dataset.__setitem__(self, key, value)
   1598     if isinstance(value, Dataset):
   1599         raise TypeError(
   1600             "Cannot assign a Dataset to a single key - only a DataArray or Variable "
   1601             "object can be stored under a single key."
   1602         )
-> 1603     self.update({key: value})
   1605 elif utils.iterable_of_hashable(key):
   1606     keylist = list(key)

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/dataset.py:5617, in Dataset.update(self, other)
   5581 def update(self, other: CoercibleMapping) -> Self:
   5582     """Update this dataset's variables with those from another dataset.
   5583 
   5584     Just like :py:meth:`dict.update` this is a in-place operation.
   (...)
   5615     Dataset.merge
   5616     """
-> 5617     merge_result = dataset_update_method(self, other)
   5618     return self._replace(inplace=True, **merge_result._asdict())

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/merge.py:1075, in dataset_update_method(dataset, other)
   1072             if coord_names:
   1073                 other[key] = value.drop_vars(coord_names)
-> 1075 return merge_core(
   1076     [dataset, other],
   1077     priority_arg=1,
   1078     indexes=dataset.xindexes,
   1079     combine_attrs="override",
   1080 )

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/merge.py:711, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
    708 skip_align_objs = [(pos, objects.pop(pos)) for pos in skip_align_args]
    710 coerced = coerce_pandas_values(objects)
--> 711 aligned = deep_align(
    712     coerced, join=join, copy=False, indexes=indexes, fill_value=fill_value
    713 )
    715 for pos, obj in skip_align_objs:
    716     aligned.insert(pos, obj)

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/alignment.py:946, in deep_align(objects, join, copy, indexes, exclude, raise_on_invalid, fill_value)
    943     else:
    944         out.append(variables)
--> 946 aligned = align(
    947     *targets,
    948     join=join,
    949     copy=copy,
    950     indexes=indexes,
    951     exclude=exclude,
...
    477         f"cannot reindex or align along dimension {dim!r} "
    478         f"because of conflicting dimension sizes: {sizes!r}" + add_err_msg
    479     )

ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1)

Anything else we need to know?

No response

Environment

Latest main xCDAT

The text was updated successfully, but these errors were encountered:

tomvothecoder · 2024-03-19T18:26:49Z

Have you come across this issue @lee1043?

lee1043 · 2024-03-19T19:24:31Z

@tomvothecoder this is interesting. No, I haven't. Maybe because my data mostly have [time, lat, lon] or [lat, lon] dimensions.

tomvothecoder · 2024-03-19T22:07:54Z

@tomvothecoder this is interesting. No, I haven't. Maybe because my data mostly have [time, lat, lon] or [lat, lon] dimensions.

It could be that the logic in _regrid expects a certain order of dimensions, or it doesn't consider an order of [lat, lev, lon].

As a path forward around this issue, I decided to use xESMF with conservative_normed instead. I got very close results when comparing to CDAT's regrid2 for all of the datasets I tested (PR).

Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 6 / 24480 (0.0245%)
Max absolute difference: 4.56288177e-07
Max relative difference: 5.34042702e-05
 x: array([[ 0.784679,  0.796702,  0.792179, ...,  0.804814,  0.773629,
         0.774743],
       [ 1.379984,  1.417249,  1.4173  , ...,  1.432473,  1.391159,...
 y: array([[ 0.784679,  0.796702,  0.792179, ...,  0.804814,  0.773629,
         0.774743],
       [ 1.379984,  1.417249,  1.4173  , ...,  1.432473,  1.391159,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc
    * var_key: OMEGA
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 3 / 24480 (0.0123%)
Max absolute difference: 5.97006945e-07
Max relative difference: 1.49039703e-05
 x: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ],
       [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584],
       [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],...
 y: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ],
       [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584],
       [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 6120 (0.0327%)
Max absolute difference: 3.39774408e-07
Max relative difference: 0.00015039
 x: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
 y: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc
    * var_key: U
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 24480 (0.00408%)
Max absolute difference: 8.52794937e-08
Max relative difference: 1.73621834e-05
 x: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 ,
        10.278145],
       [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,...
 y: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 ,
        10.278145],
       [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc
    * var_key: U
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 6120 (0.0163%)
Max absolute difference: 1.45889873e-07
Max relative difference: 1.08735566e-05
 x: array([[ 9.712424,  9.692257,  9.675111, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
 y: array([[ 9.712424,  9.692257,  9.675112, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc
    * var_key: U
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 24480 (0.00408%)
Max absolute difference: 8.38617442e-08
Max relative difference: 1.96239722e-05
 x: array([[ 8.311289,  8.302058,  8.292533, ...,  8.338692,  8.329687,
         8.320503],
       [10.096993, 10.085039, 10.072245, ..., 10.131809, 10.120227,...
 y: array([[ 8.311289,  8.302058,  8.292533, ...,  8.338692,  8.329687,
         8.320503],
       [10.096994, 10.085039, 10.072245, ..., 10.131809, 10.120227,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 9792 (0.0204%)
Max absolute difference: 3.15701861e-07
Max relative difference: 2.42041494e-05
 x: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962],
       [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424],
       [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],...
 y: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962],
       [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424],
       [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc
    * var_key: OMEGA
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 9792 (0.0204%)
Max absolute difference: 6.93198768e-07
Max relative difference: 5.40329309e-05
 x: array([[ 1.283442,  1.27651 ,  1.25776 , ...,  1.292833,  1.29537 ,
         1.287421],
       [ 1.93963 ,  1.950937,  1.957762, ...,  1.901146,  1.91872 ,...
 y: array([[ 1.283442,  1.27651 ,  1.25776 , ...,  1.292833,  1.29537 ,
         1.287421],
       [ 1.93963 ,  1.950937,  1.957762, ...,  1.901146,  1.91872 ,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc
    * var_key: OMEGA
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 6120 (0.0327%)
Max absolute difference: 3.39774408e-07
Max relative difference: 0.00015039
 x: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
 y: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc
    * var_key: U
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 9792 (0.0102%)
Max absolute difference: 1.06174015e-07
Max relative difference: 1.43005542e-05
 x: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217,
        10.64816 ],
       [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,...
 y: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217,
        10.64816 ],
       [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc
    * var_key: U
    
Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 6120 (0.0163%)
Max absolute difference: 1.45889873e-07
Max relative difference: 1.08735566e-05
 x: array([[ 9.712424,  9.692257,  9.675111, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
 y: array([[ 9.712424,  9.692257,  9.675112, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)

jasonb5 · 2024-03-19T22:28:10Z

I'll take a look this week, the code assumes the dimension ordering is [..., X, Y], I'll adjust this to be more flexible.

tomvothecoder · 2024-03-20T16:26:34Z

@jasonb5 Sounds good, thanks.

tomvothecoder · 2024-03-28T22:47:28Z

@jasonb5 Unless this can be resolved by next Tues, we can push it back to the next release. I don't think it is a high priority because Jiwoo hasn't ran into it and I'm using xESMF in e3sm_diags instead.

jasonb5 · 2024-04-05T02:02:42Z

@tomvothecoder Go ahead and push this out to the next release.

tomvothecoder · 2024-04-08T16:03:22Z

@jasonb5 Sounds good

tomvothecoder added the type: bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Mar 19, 2024

tomvothecoder assigned jasonb5 Mar 19, 2024

tomvothecoder changed the title ~~[Bug]: Regrid2 _regrid incorrectly transposes output_data shape, which doesn't align with original dim order~~ [Bug]: Regrid2 _regrid incorrectly transposes output_data shape, which doesn't align with original dim order used by bounds Mar 19, 2024

tomvothecoder mentioned this issue Mar 19, 2024

CDAT Migration: Refactor meridional_mean_2d set E3SM-Project/e3sm_diags#795

Merged

9 tasks

tomvothecoder added this to the FY24Q2 (01/01/24 - 03/31/24) milestone Mar 20, 2024

jasonb5 mentioned this issue Apr 5, 2024

[PR]: Update Regrid2 missing and fill value behaviors to align with CDAT and add unmapped_to_nan arg for output data #613

Merged

9 tasks

tomvothecoder modified the milestones: FY24Q2 (01/01/24 - 03/31/24), FY24Q3 (4/01/24 - 06/30/24) Apr 10, 2024

jasonb5 mentioned this issue May 9, 2024

Fixes regrid2 mapping output to input ordering #653

Merged

9 tasks

tomvothecoder closed this as completed in #653 May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

tomvothecoder commented Mar 19, 2024 •

edited

tomvothecoder commented Mar 19, 2024

lee1043 commented Mar 19, 2024

tomvothecoder commented Mar 19, 2024 •

edited

jasonb5 commented Mar 19, 2024

tomvothecoder commented Mar 20, 2024

tomvothecoder commented Mar 28, 2024

jasonb5 commented Apr 5, 2024

tomvothecoder commented Apr 8, 2024

[Bug]: Regrid2 breaks in _preserve_bounds because _regrid incorrectly transposes output_data shape, which doesn't align with original dim order used by bounds #625

[Bug]: Regrid2 breaks in _preserve_bounds because _regrid incorrectly transposes output_data shape, which doesn't align with original dim order used by bounds #625

Comments

tomvothecoder commented Mar 19, 2024 • edited

What happened?

Example

What did you expect to happen? Are there are possible answers you came across?

Minimal Complete Verifiable Example (MVCE)

Relevant log output

Anything else we need to know?

Environment

tomvothecoder commented Mar 19, 2024

lee1043 commented Mar 19, 2024

tomvothecoder commented Mar 19, 2024 • edited

jasonb5 commented Mar 19, 2024

tomvothecoder commented Mar 20, 2024

tomvothecoder commented Mar 28, 2024

jasonb5 commented Apr 5, 2024

tomvothecoder commented Apr 8, 2024

[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

tomvothecoder commented Mar 19, 2024 •

edited

tomvothecoder commented Mar 19, 2024 •

edited