[BUG/ISSUE] Invalid time index in daily data organized in monthly files if HEMCO is not called at midnight of last day every month

[jimmielin]

Report a HEMCO bug or technical issue

This bug was discovered by Xu Feng (@fengx7) when running WRF-GC, but this could happen in any model using HEMCO. A detailed autopsy is below.

Scenario

• A series of emissions / data updated daily are stored in monthly files. This happens in GFED daily scale factors, e.g.:

(((GFED_daily
111 GFED_FRAC_DAY   $ROOT/GFED4/v2020-02/$YYYY/GFED4_dailyfrac_gen.025x025.$YYYY$MM.nc GFED_FRACDAY 2003-2019/1-12/1-31/0  RF xy 1 * - 1 1
)))GFED_daily

• At the last month of the day, HEMCO is not called at midnight. This is not a typical scenario in GEOS-Chem, but assume a model calling HEMCO runs on 7-minute time steps, the time stepping would then look like:

2019-10-30 00:00:00
2019-10-30 00:07:00
...
2019-10-30 23:55:00
**2019-10-31 00:02:00**
2019-10-31 00:09:00

HEMCO will crash in this scenario saying Invalid time index for data at 2019-10-31 00:02:00.

Why is this happening?

The HEMCO verbose log shows that:

Set HEMCO clock to 2019-10-31 00:02:00
The weekday is (0=Sun,...,6=Sat): 4

HEMCO: Opening GFED4_daily_025x025.201910.nc
HEMCO: Entering GET_TIMEIDX (hco_read_std_mod.F90) ( 6)
 Number of time slices found:           31
 Time slice range :    201910010000.000        201910310000.000
          preferred datetime:  201910310002.
              selected tidx1:             -1
        assigned delta t [h]:              0
 local time?  F
HEMCO: Leaving GET_TIMEIDX (hco_read_std_mod.F90) ( 6)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Invalid time index in GFED4_daily_025x025.201910.nc

In this case, HEMCO was called not at midnight. It attempts to search for 201910310002.. Even though this is daily data, HEMCO first checks the time slice range and finds that 201910310002. is after the last time slice available in the file, 20191031000.000, and thus crashes.

The expected behavior is that HEMCO will read 201910310000. because this is daily data and the hour/minute does not matter.

This does not crash at other days because otherwise the current HEMCO clock still lies between 00:00 of the first month and 00:00 of the last day. e.g., if 201910300002. is the current HEMCO clock, HEMCO will correctly recognize that this is within the range 201910010000. and 201910310000., and select the closest time slice (201910300000.). There is no drift.

Proposed solution

In this case HEMCO should recognize that 201910310000.000 is a valid time slice for the daily-updated data requested at time step 201910310002. I think a fix to the range check would be OK, because if that happens HEMCO will correctly find the closest time slice.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.