Issue reading FGOALS-f3-L Omon/thetao CMIP netcdf that is (S3) cloud-resident

[aradhakrishnanGFDL]

Description

intake_esm works great with CMIP6 netcdf output in S3. A recent issue though is related to reading the following using intake_esm. Interestingly, a copy of the exact same datasets in posix opens fine using intake_esm. 

s3://esgf-world/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Omon/thetao/gn/v20191007/thetao_Omon_FGOALS-f3-L_historical_r1i1p1f1_gn_190001-194912.nc
s3://esgf-world/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Omon/thetao/gn/v20191007/thetao_Omon_FGOALS-f3-L_historical_r1i1p1f1_gn_185001-189912.nc
s3://esgf-world/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Omon/thetao/gn/v20191007/thetao_Omon_FGOALS-f3-L_historical_r1i1p1f1_gn_195001-201412.nc

What I Did

I tried to use intake_esm with the [catalog](col_url = "https://cmip6-nc.s3.us-east-2.amazonaws.com/esgf-world.json"
) to open the above mentioned datasets. Few other models used in the analysis seem to open fine, but an issue crops up when we attempt to use the data mentioned above.

The notebook with the code to reproduce the issue and info on the versions -- can be found here

I also tried using the following which throws the same error

import s3fs
import xarray as xr
%matplotlib inline

fs_s3 = s3fs.S3FileSystem(anon=True)
s3path = 's3://esgf-world/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Omon/thetao/gn/v20191007/thetao_Omon_FGOALS-f3-L_historical_r1i1p1f1_gn_190001-194912.nc'

remote_file_obj = fs_s3.open(s3path, mode='rb')

ds = xr.open_dataset(remote_file_obj, chunks={'time': 1})

ds

cftime: 1.2.1
dask: 2.30.0
fastprogress: 0.2.7
fsspec: 0.8.4
gcsfs: 0.7.1
intake: 0.6.2
intake_esm: 2021.1.15
netCDF4: 1.5.6
pandas: 1.1.4
requests: 2.24.0
s3fs: 0.4.2
xarray: 0.17.0
zarr: 2.5.0

Your help is much appreciated. Please let me know if I missed anything!

[aradhakrishnanGFDL]

Just a note if it helps- Randomly checking a few 2D Omon fields from the same model/experiment, seems to work okay.
In addition to thetao, another 3D field (so) also shows the same error and it has three assets as well, each 50 years I think
example-
s3://esgf-world/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Omon/so/gn/v20191007/so_Omon_FGOALS-f3-L_historical_r1i1p1f1_gn_185001-189912.nc