LIGHT hist files miss some days around month-end transition

[bradyrx]

I've noticed that some of the lagrPartTrack hist files from LIGHT are missing anticipated days on the 1st of some months. We did a 30to10 BGC run with online LIGHT particles. The LIGHT particles were output at 2 day hist frequencies.

For instance, we ran particles from 0034-0050 for this run. Anticipated xtime for the particles would be:

import xarray as xr
expected_dates = xr.cftime_range(start='0034-01-02', end='0050-12-30', freq='2D', calendar='noleap')

I then constructed an xtime list from the actual particle output:

import cftime
import numpy as np
import xarray as xr
def extract_time(t):
    # Converts xtime into a cftime object.
    y, m, d = t.split('-')
    d = d.split('_')[0]
    return cftime.DatetimeNoLeap(int(y), int(m), int(d), 0, 0, 0)

actual_dates = np.array([])
# filelist is the list of all lagrPartTrack hist files
for f in tqdm(filelist):
    ds = xr.open_dataset(f)
    times = ds.xtime.values.astype(str)
    temporary_dates = np.array([extract_time(t) for t in times])
    actual_dates = np.hstack([dates, temporary_dates])

We can then look at what anticipated days are missing from LIGHT:

actual_dates = xr.CFTimeIndex(actual_dates)
actual_dates = set(np.asarray(actual_dates))
expected_dates = set(np.asarray(expected_dates))

We're missing 27 expected time steps. Fortunately, it seems like a systematic issue that only occurs on the first of each month:

print(actual_dates ^ expected_dates)

>>> {cftime.DatetimeNoLeap(34, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(35, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(36, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(37, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(38, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(39, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(41, 5, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(43, 8, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(44, 2, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(44, 6, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(44, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(45, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(46, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(46, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(47, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(47, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(47, 12, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(48, 3, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(48, 6, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(48, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(48, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(49, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(49, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(49, 11, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(50, 2, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(50, 6, 1, 0, 0, 0, 0),
 cftime.DatetimeNoLeap(50, 9, 1, 0, 0, 0, 0)}

Due to the high resolution of the Eulerian run, we frequently had run cycles that were less than one year, ending up with monthly restart files mid-year. My guess is it occurs when a particle hist file saves out N-1 days for a given month and coincides with the end of a run cycle.

For instance, 0034-06-29 was the last particle save for June. I don't have access to the restart files, but maybe we ended the run that month. I believe the particle locations would then be saved out for midnight following June 30th, i.e. July 1st, which is read in as a restart file for the next cycle and then the first hist date we get is 0034-07-03 for that run.

CC @pwolfram, @maltrud. I'm aware LIGHT might not be under active development anymore, although if anyone else picks it up this could be useful. Or maybe this is the reality of running an analysis member like this with clashing hist frequencies (5daily/monthly for Eulerian and 2daily for LIGHT).

[bradyrx]

My analysis solution is just to have my particle dt be 3 days or 4 days, depending on month length, for the missing dates. Not sure if there's any way to fix this at run time.

[xylar]

@bradyrx, with @pwolfram not working on MPAS anymore, it's pretty unlikely that we will have a chance to look at this anytime soon. On the other hand, it sounds more likely to be a timekeeping issue generic to all AMs that might come up in another context. We do definitely tend to run our AMs in tact with our model restarts so it may be something we just haven't noticed.

Anyway, seems really annoying.

[bradyrx]

@bradyrx, with @pwolfram not working on MPAS anymore, it's pretty unlikely that we will have a chance to look at this anytime soon. On the other hand, it sounds more likely to be a timekeeping issue generic to all AMs that might come up in another context. We do definitely tend to run our AMs in tact with our model restarts so it may be something we just haven't noticed.

Anyway, seems really annoying.

Yeah I figured that was the case re: LIGHT. But that this might be something that could come up with other AMs. The key thing here is if there's a mis-match in the AM timescale (here, 2 days) and on the model hist timescale (here, 5 days or 1 month) then you can miss some time steps.