Fix results from the Python/bash pipeline

[griff-rees]

We were able to transfer the results but found a bug in the date range (I think that was the core @RuthBowyer? sorry if I'm misremembering). Below are components to check that may have a played a role as best I can remember:

• Pre-processing (see the pipeline assuming that's up to date)
  • Differences in start dates in python elements
  • Temporal re-sampling CPM to 365 days per year (to match Hads at 365) #106
  • Paths by which these files are saved and then used in following steps
• Application of the python debiasing methods

[RuthBowyer]

Hey there - came to add to this - I believe the issue was that per year, there were not 360 days in the corrected dataset. This might be fixed anyway when we update the resampling as in #106 - but it could have been introduced at a later stage?

[andrewphilipsmith]

Note (for future investigation):

In python/resampling/resampling_hads.py we call

data_360 = data.convert_calendar(dim="time", calendar="360_day", align_on="year")

followed by

# apply correction if leap year
if data.time.dt.is_leap_year.any():
    data_360 = enforce_date_dropping(data, data_360)

However in python/load_data/data_loader.py we just call without calling enforce_date_dropping:

x = x.convert_calendar(dim="time", calendar="360_day", align_on="year")

[griff-rees]

Thanks @andrewphilipsmith. Just for continuity since comments added in #122, the enforce_date_dropping function is written within python/resampling/resampling_hads.py and seems a local attempt to address pydata/xarray#8086 which was raised in relation to #32.

[griff-rees]

@RuthBowyer, do you know the details of the method used on the R side, and whether it's dropping days (which I think is what enforce_date_dropping is designed to do) versus a decimal interpolation like interp_calendar in xarray? I'm guessing this relates to #106 and #32 at least.

[RuthBowyer]

Hey there
So sorry for the lack of clarity on this. The 'R pipeline' only ever had code to resample the data to the same grid, not calendar days - because when we were at this stage during our initial workings we were focused on the whole of the UK, we decided to only do the resampling of the calendar days in python, and then use this input into the R work. I'm so sorry this is so unclear and messy! The original plan was to have a single pre-processing pipeline that then had data useable in both R and python, but that's not how it all worked out, apologies!
When we realised the calendar day problem, for the previous R running of everything, I just did a 'hacky' fix of the days and, as at that time the dataset had too many days in leap years just hard coded the removal of the extra days in those years.