Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating hubDocs to note hubUtils split into hubAdmin and hubData #101

Merged
merged 4 commits into from Mar 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/source/quickstart-hub-admin/uploading-validating.md
Expand Up @@ -15,12 +15,12 @@ Once you have uploaded those two files onto your GitHub repository, you will not

## Validating your config files

Now you should validate the config files, to be sure they are properly functional. You can use the `validate_config` function from [`hubUtils`](https://infectious-disease-modeling-hubs.github.io/hubUtils/index.html) to check whether Hub config files are valid. The steps are as follows:
Now you should validate the config files, to be sure they are properly functional. You can use the `validate_config` function from [`hubAdmin`](https://infectious-disease-modeling-hubs.github.io/hubAdmin/index.html) to check whether Hub config files are valid. The steps are as follows:

1. First, you need to install the package `hubUtils`. [Here are instructions for installing `hubUtils` in the R console](https://github.com/Infectious-Disease-Modeling-Hubs/hubUtils#installation).
2. Next, you need to validate the config files. [Here are detailed instructions and explanations](https://infectious-disease-modeling-hubs.github.io/hubUtils/articles/hub-setup.html#validate-config-files), and following is sample code you can copy to validate the config files:
1. First, you need to install the package `hubAdmin`. [Here are instructions for installing `hubAdmin` in the R console](https://github.com/Infectious-Disease-Modeling-Hubs/hubAdmin#installation).
2. Next, you need to validate the config files. [Here are detailed instructions and explanations](https://infectious-disease-modeling-hubs.github.io/hubAdmin/articles/hub-setup.html#validate-config-files), and following is sample code you can copy to validate the config files:
```
library(hubUtils)
library(hubAdmin)

validate_config(
hub_path = ".",
Expand Down
2 changes: 1 addition & 1 deletion docs/source/quickstart-hub-admin/using-hub.md
Expand Up @@ -31,7 +31,7 @@ This technical ReadMe should be stored on your hub. You will need to share this
Before you put out the word that you are accepting submissions of models (such as forecasts) and before you give a date of when the submissions are due, you should be prepared to accept the data. The [`hubValidations`](https://infectious-disease-modeling-hubs.github.io/hubValidations/) repository facilitates the implementation of general validation rules that are enforced on submissions in the form of [pull requests](https://docs.github.com/articles/about-pull-requests) to hub repositories.

## Looking at the data
Once you have collected some models, you will want to explore the data. [`hubUtils`](https://infectious-disease-modeling-hubs.github.io/hubUtils/) provides a set of utility functions for downloading, plotting, and scoring models/forecasts and truth data from modeling hubs.
Once you have collected some models, you will want to explore the data. We have built a [software suite](../user-guide/software.md) to support common modeling hub tasks, like loading model output data, plotting the model output data, building ensembles using the data, and in some cases evaluating the predictions made by different models.

## Building ensembles
[`hubEnsembles`](https://infectious-disease-modeling-hubs.github.io/hubEnsembles/) is an `R` package with functionality to build simple ensembles of data from modeling hubs. Different ensembles can be built using for instance the mean, median, or mode.
Expand Down
3 changes: 2 additions & 1 deletion docs/source/user-guide/model-metadata.md
Expand Up @@ -22,11 +22,12 @@ Many hubs will use a common set of metadata fields, which we place in the follow

## Template metadata schema file

The following is a [*template metadata schema file* from the **Example Complex Forecast Hub**](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/blob/main/hub-config/model-metadata-schema.json), which is itself drawn from the [**FluSight Forecast Hub**](https://github.com/cdcepi/FluSight-forecast-hub/blob/main/model-metadata/README.md). Many Hubs will use the same fields listed in the *template metadata schema file*, but some may need fewer or more. Administrators can remove fields as needed, and can use [additional optional metadata fields from this list](#optional-additional-metadata-fields). We encourage administrators to use the same standard names and definitions to maintain consistency across Hubs.
The following is a [*template metadata schema file* from the **Example Complex Forecast Hub**](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/blob/main/hub-config/model-metadata-schema.json), which is itself drawn from the [**FluSight Forecast Hub**](https://github.com/cdcepi/FluSight-forecast-hub/blob/main/model-metadata/README.md). Many Hubs will use the same fields listed in the *template metadata schema file*, but some may need fewer or more. A Hub's *model metadata schema file* should include either the single combined field `model_id`, or both of the fields `team_abbr` and `model_abbr`. Administrators can remove other fields as needed, and can use [additional optional metadata fields from this list](#optional-additional-metadata-fields). We encourage administrators to use the same standard names and definitions to maintain consistency across Hubs.

<script src="../_static/docson/widget.js" data-schema="https://raw.githubusercontent.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/main/hub-config/model-metadata-schema.json"></script>


(optional-additional-metadata-fields)=
## Optional additional metadata fields
Depending on the goals and context of a particular Hub, it may be beneficial to collect more information about the modeling assumptions made by models. The following fields have been included within the model_details metadata field in past Hubs. Where applicable, we recommend that hubs draw from this list to encourage standardization of terminology across Hubs. If Hubs develop new model metadata items, they should be added to this list for future reference.

Expand Down
10 changes: 9 additions & 1 deletion docs/source/user-guide/software.md
Expand Up @@ -2,9 +2,17 @@

To assist users in building a hub, we have developed a software suite with specific functions and uses outlined below. These tools are designed to support common modeling hub tasks, like loading model output data, plotting the model output data, building ensembles using the data, and in some cases evaluating the predictions made by different models.

## [`hubData`](https://github.com/Infectious-Disease-Modeling-Hubs/hubData)

The goal of `hubData` is to provide tools for connecting to, interacting with, and manipulating Hub data. [Here are instructions](https://github.com/Infectious-Disease-Modeling-Hubs/hubData) to download and use the package.

## [`hubAdmin`](https://github.com/Infectious-Disease-Modeling-Hubs/hubAdmin)

The `hubAdmin` package contains utility functions for administering Hubs, in particular creating and validating hub configuration files. [Here are instructions](https://github.com/Infectious-Disease-Modeling-Hubs/hubAdmin) to download and use the package.

## [`hubUtils`](https://infectious-disease-modeling-hubs.github.io/hubUtils/)

The goal of `hubUtils` is to provide a set of utility functions for downloading, plotting, and scoring models (e.g., forecasts) and truth data from Infectious Disease Modeling Hubs. You can find instructions to download and use the package [here](https://infectious-disease-modeling-hubs.github.io/hubUtils/).
`hubUtils` is a lightweight package that primarily contains general utilities imported by other hubverse packages. Previously, `hubUtils` was a larger package with more functions, but most of these were moved and split across `hubData` and `hubAdmin`.

## [`hubEnsembles`](https://github.com/Infectious-Disease-Modeling-Hubs/hubEnsembles)

Expand Down
6 changes: 3 additions & 3 deletions docs/source/user-guide/tasks.md
Expand Up @@ -30,11 +30,11 @@ The figure shows that Hub metadata and target data are specified by the hub itse

Task ID variables can be thought of as columns of a tabular representation in a model output file, where a combination of values of task ID variables would uniquely define a row of data.

We note that some task ID variables are special in that they conceptually define a modeling "target" (these are referred to in the [tasks metadata](tasks_metadata) as a `target_key`). In our [Running Example 1](running-examples), the task ID variables are `target`, `location`, and `origin_date`. In this example, `target` is the target key and can only take on one value "inc covid hosp". In other examples, (such as [Running Example 3](running-examples)) more than one variable can serve as target keys together. In example 3,both 'outcome_variable' and 'outcome_measure' make up the target keys.
We note that some task ID variables are special in that they conceptually define a modeling "target" (these are referred to in the [tasks metadata](tasks_metadata) as a `target_key`). In our [Running Example 1](running-examples), the task ID variables are `target`, `location`, and `origin_date`. In this example, `target` is the target key and can only take on one value "inc covid hosp". In other examples, (such as [Running Example 3](running-examples)) more than one variable can serve as target keys together. In example 3,both 'outcome_variable' and 'outcome_measure' make up the target keys.

Some task ID variables serve specific purposes. For example, every hub must have a single task ID variable that uniquely defines a submission round. It has become a convention to use a task ID like `origin_date` or `forecast_date` for this purpose, although in practice hubs could use other task ID variables for this purpose. In [Running Example 1](running-examples), this task ID is `origin_date`. Additionally, to plot forecasts using standard functionality in the `hubUtils` package for hubs with step-ahead short term predictions, hubs must have either `origin_date` and `horizon` task ID or a `target_date` task ID.
Some task ID variables serve specific purposes. For example, every hub must have a single task ID variable that uniquely defines a submission round. It has become a convention to use a task ID like `origin_date` or `forecast_date` for this purpose, although in practice hubs could use other task ID variables for this purpose. In [Running Example 1](running-examples), this task ID is `origin_date`.

In general, there are no restrictions on what task ID variables may be named, however when appropriate, we suggest that Hubs adopt the following standard task ID or column names and definitions:
In general, there are no restrictions on what task ID variables may be named, however when appropriate, we suggest that Hubs adopt the following standard task ID or column names and definitions:

* `origin_date`: the starting point that can be used for calculating a target_date via the formula `target_date` = `origin_date` + `horizon` * `time_units_per_horizon` (e.g., with weekly data, `target_date` is calculated as `origin_date` + `horizon` * 7 days).
* `forecast_date`: usually defines the date that a model is run to produce a forecast.
Expand Down