Converting archived hubs to fully-configured hubVerse-formatted hubs

[lmullany]

We are going to try to convert one or more achived/no-long-active hubs to hubverse format. The first one that we will attempt to do this for is the 2015-2019 FluSight hub.

[bsweger]

@lmullany Sounds great! @nickreich and I were talking about how this effort ties in nicely to the ongoing infrastructure work for cloud-enabled hubs.

Because there are no active submissions to worry about, we could use the data you're creating to test the cloud sync and parquet conversions without the worry of disrupting a live hub.

Pushing this historic data to the cloud would also provide a tangible talking point for our conversations about cloud data access, optimization, etc

Very excited!

[lmullany]

@bsweger @nickreich: the initial version of the hubverse-formatted flusight1 repo is here:

https://github.com/lmullany/flusight1_hub

This should be considered preliminary. There are a number of issues that we might want to sort through including:

1. Is the selection of the origin_epiweek, as the main round identifier, the right choice?
2. How should we handle in-valid data? For example, we have model submissions where the cdf max exceeds 1, or the pmf does not sum to 1
3. Some teams changed their name from year to year, and thus in model-output can have multiple folders, but if this is truly a name change only, perhaps a common name should be chosen and the multiple folders consolidated into on
4. There is nothing in model-metadata folder, because the original repo did not contain a metadata information file for each of the models.
5. Occaisionally teams submitted two .csv files for the same EWXX value. For example, we can have something like EW50_Team_model_2019-12-25.csv and EW50_Team_model_2019-12-27.csv. In these situations, I took the earliest file (i.e. the EW50 file with the 2019-12-25 date embedded in the file name

Maybe we can setup a time to discuss further

[nickreich]

Thanks for the progress on this! I don't have access to the repo. I think some of the issues you list might be more thorny than others. I might suggest making separate issues in the repo you created for each of the above issues and then we could have threaded conversations about each of them.

[lmullany]

Will do

[lmullany]

@nickreich I've added issues to the github for the above concerns

[bsweger]

Very exciting--thanks, @lmullany !

I'll await the more specific issues before chiming in, but in the meantime, I'm continuing to codify the function that converts model-output data to parquet (add adds columns to round_id, team, and model) so it will trigger automatically when a hub's data is synced to the cloud.

This dovetails nicely with the work you're doing. Once we have an official hubverse-compatible version of the flusight archive, we can add the cloud infrastructure pieces to make the data accessible via S3. 🚀

[lmullany]

Here, team and model are not well distinguished. While model metadata in a “typical” repo that is created ( and therefore validated) from the outset using hubverse tools require this distinction ( I think!), that may not necessarily be true with post hoc created repo like this one.

So we might want to consider designing the functionality you mention above with some degree of flexibility with regards to the structure of model-output data and the presence/absence of information in the model-metadata folder

[bsweger]

Interesting--thanks for the heads-up.

The transform function's unit of work is a single model-output file, which allows us to use S3 event notifications + AWS Lambda to transform each model-output file after it arrives in the bucket.

Your comment is a good note to parse the team and model name via directory structure and not filename (the S3 event sends the entire object key, so that's not a problem). Appreciate it!

Infectious-Disease-Modeling-Hubs/hubverse-transform#2

[lmullany]

@bsweger the current version of the repo is structured like this:

Note that model-output folder has a subfolder for each model (the name of that folder is sometimes in team-model format (for example "NEU-GLEAM", but in many instances that type of information was not provided, so we have folder names that really just describe the model itself

Also note that within the sub-folder, we have a list of parquet files, each one labeled as "YYYY-EW", where EW is the epiweek. This 7 character string is also the round_id. (That column is included in the parquet file as well as a constant).

Does the constant column have to be included?.. Or can the tasks.json be set up to so that the round_id is taken not from variable, but from filename?

[LucieContamin]

please feel free to transfer it

[LucieContamin]

Yeah, so I guess I would vote for resolving the epiweeks to an origin_date column then, for simplicity. Maybe we keep an epiweek column, for posterity?

Sorry I misunderstood! I agree that it might be helpful to add a date column and maybe keep the epiweek column so you don't need to change the round_id. Will that work?

[lmullany]

Most of all of this above belongs here: lmullany/flusight1_hub#10

I'm going to move forward with creating a new column origin_date, and have started a new issue here to discuss the exact derivation of that origin_date

[nickreich]

Some quotes from the 2017-2018 FluSight guidance docs:

Forecasts for the ILINet percentage for the four weeks following the forecast submission should be relative to the most recent week of ILINet data released. For example, ILINet data for week 43 will be posted on Friday, November 3 at 12:00PM Eastern Time. The four-week forecast submitted on Monday, November 6 should include predictions for ILINet values for weeks 44-47.

and

the filename should be modified to the following standard naming convention: a forecast submission using week 43 surveillance data submitted by John Doe University on November 6, 2017, should be named “EW43-JDU-2017-11-06.csv” where EW43 is the latest week of ILINet data used in the forecast, JDU is the name of the team making the submission (e.g. John Doe University), and 2017-11-06 is the date of submission.

I just wanted to clarify/confirm that the submission date is going to be used as the "round_id", and that we discussed yesterday that we might standardize this on being something like the last day (Saturday) of the epiweek prior to the Monday submission deadline? Whereas the origin date will likely be a week or so behind because of the ILI delays. see the issue @lmullany mentioned above for more details

[lmullany]

Actually, we are going to use the Saturday of the EWXX that is the prefix of each submission file, regardless of the submission date embedded in the filename.

So, this is basically in line with what you have here, but I want to be explicit. This is also option 2 in the three options I laid out in the issue you mention above:

lmullany/flusight1_hub#19

[annakrystalli]

Comment Moved to thread

[lmullany]

Regarding the submission due property, I don't think it's a problem that retrospectively some submissions may not have met the submission window criteria. Indeed, even in current hubs, a mantainer may allow a submission outside the window to be merged in if they like.

Having said that, moving to a date round_id and setting up a relative submission would mean that, in theory, if you ever wished to open the hub up for submissions, all you would need to do is add to the optional values of origin_date and submission windows for new round_ids would be validated successfully. It would also serve as an example of an actually functional and correctly configured hub.

I tend to now think though that a more important reason to convert to dates might be downstream functionality like scoring or visualising hub data but I'll let folks that are more familiar with those aspects (e.g. @elray1 , @LucieContamin @nickreich etc) weigh in on that.

In any case, if you do decide to go ahead with it, I found this snippet useful when playing around with configs for this hub:

epiweek <- "2022-52"
lubridate::parse_date_time(paste(epiweek, 1, sep = "-"), 'Y-W-w')
#> [1] "2022-12-26 UTC"

Created on 2024-04-26 with reprex v2.0.2

fyi @annakrystalli , I'm not sure that snippet is correct for convert epi weeks to a date. I use the following for that. Perhaps there is a better built in function that I'm not aware of.

#' Get start date of (epi) given epiweek and epiyear
#' 
#' Function returns the date of the first day of the epiweek defined by
#' user provided epiweek (`ew`) and epiyear (`ey`)

date_from_epiweekyear <- function(ey,ew) {

  # internal function gets the start date of the first week of the 
  # give epi year
  f_epi_week_start <- function(y) {
    j4 = strptime(paste0(y, "-01-04"), "%Y-%m-%d")
    dplyr::if_else(j4$wday == 7, as.Date(j4), as.Date(j4) - j4$wday)
  }
  
  # get the start date of this epi year
  start_1 = f_epi_week_start(ey)
  
  # check that the epi week passed by user does not exceed the
  # maximum available for this epi year
  max_week = (f_epi_week_start(ey+1)-start_1)/7
  if(ew>max_week) {
    "The provided epiweek number is not valid for this year"
    break
  }
  # return the start date of this epiweek/epiyear
  return(as.Date(start_1+(ew-1)*7))
}

[lmullany]

yeah, lubridate should really offer a reverse function, otherwise you have to go to a number of other packages.. I just wanted to a quick implemention without installing any other package

[nickreich]

Agree with wanting this to be simple and the desire to not rely on outside packages. The logic you use is at least a bit different from the "standard" MMWR implementation. Do they produce the same results?

[lmullany]

I’ll check to be sure. Maybe I moved too fast and made some error. If so, I’ll fix.

[lmullany]

yep, it looks okay to me. I don't see any errors. Here is a much faster, more concise, and vectorized version that uses the same logic

date_from_epiweekyear <- function(ey,ew) {
  # internal function gets the start date of the first week of year
  f <- \(y) { j4=paste0(y, "-01-04"); as.Date(j4) - wday(j4)+1}
  # get the start date of this epi year and next epi year
  s1 = f(ey); s2=f(ey+1)
  # check that max week is not exceeded  
  data.table::fifelse(ew>(f(ey+1) - s1)/7, NA_Date_, s1+(ew-1)*7)
}

Here is a quick test over the last couple of decades:

library(data.table); library(lubridate)

df = data.table(date = seq.Date(as.Date("2001-01-01"), as.Date("2024-12-31"),1))
df[, c("ey", "ew", "ws"):={ey = epiyear(date); ew=epiweek(date);ws = date_from_epiweekyear(ey,ew); .(ey, ew,ws)}]

# Now, confirm that the ws is equal to date-wday()+1  
df[, all(ws==date-wday(date)+1)]
[1] TRUE

and, checking against MMWR week:

df[, mmwr:=MMWRweek::MMWRweek2Date(ey, ew)][, all(ws==mmwr)]
[1] TRUE

[nickreich]

awesome!

[lmullany]

Question: how does the group feel about changing model name to all lower case (or all upper case). Similarly for team name.
Examples:

• KoT-GPthingy would become kot-gpthingy
• Delphi-Archefilter would become delphi-archefilter

Not a requirement, but would look a bit more standard, and would help a bit as we are renaming / merging different misspellings, etc.

[bsweger]

yes, please!

[annakrystalli]

Definitely in favour of all lower case personally!