Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explaining mean in tasks.json #102

Merged
merged 2 commits into from Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/source/quickstart-hub-admin/tasks-config.md
Expand Up @@ -44,7 +44,7 @@ Save the file in the `hub-config` folder (which is [in your repository on your l

## Step 5: Examine the new `tasks.json` file

Open `tasks.json` and explore the content and structure. Some key concepts are defined [here](../overview/definitions.md), and additional explanations are offered below:
Open `tasks.json` and explore the content and structure. Some [key concepts are defined here](../overview/definitions.md), and [a full explanation of all the supported elements in a `tasks.json` file can be found here](https://raw.githubusercontent.com/Infectious-Disease-Modeling-Hubs/schemas/main/v2.0.1/tasks-schema.json#L50). Simple explanations for elements in the Example Forecast Hub file are offered below:

* `schema_version`: Modeling Hub [Schema](../overview/definitions.md) versions are all housed in [this repository](https://github.com/Infectious-Disease-Modeling-Hubs/schemas/).
* `round_id`: The [round](../overview/definitions.md) identifier establishes which date from a forecast submission is used to identify the submission round it corresponds to (e.g., the origin date).
Expand Down Expand Up @@ -98,11 +98,11 @@ Now, read below for details on some of the lines of code in this file:
```

### 5.5. Defining [`"output_type"`](#model_output_format):
- The [`output_type`](#model_output_format) is used to establish the valid model output types for a given modeling task. In this example they include `mean` and `quantile`.
- The [`output_type`](#model_output_format) is used to establish the valid model output types for a given modeling task. In this example they include `mean` and `quantile`, but `median`, `cdf`, `pmf`, and `sample` are other supported output types. Output types have two additional properties, an `output_type_id` and a `value` property, both of which establish the valid values that can be entered for this output type.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a request for a change here, but flagging that based on ongoing work on sample output types, we might eventually change this so that there would be an output_type_id_params argument in there as well.


#### 5.5.1. Setting the `"mean"`:
- <mark style="background-color: #FFE331">Here, the `"mean"` of the predictive distribution</mark> is set as a valid value for a submission file.
- <mark style="background-color: #32E331">`"output_type_id"` is used</mark> to determine whether the `mean` is a required or an optional `output_type`. In this example, the mean is optional, not required. If the mean is included, the `"output_type_id"` should be NA, since `"output_type_id"` is not used for mean predictions.
- <mark style="background-color: #32E331">`"output_type_id"` is used</mark> to determine whether the `mean` is a required or an optional `output_type`. Both `"required"` and `"optional"` should be declared, and the option that is chosen (required or optional) should be set to `["NA"]`, whereas the one that is not chosen, should be set to `null`. In this example, the mean is optional, not required. If the mean is required, `"required"` should be set to `["NA"]`, and `"optional"` should be set to `null`.
- <mark style="background-color: #38C7ED">`"value"` sets the characteristics</mark> of this valid `output_type` (i.e., the mean). In this instance, the value must be an `integer` greater than or equal to `0`.

```{image} ../images/tasks_schema_5-1.png
Expand Down
6 changes: 3 additions & 3 deletions docs/source/user-guide/hub-config.md
Expand Up @@ -2,9 +2,9 @@

## Directory Structure
The `hub-config` directory in a modeling hub is required to contain three files:
1. `admin.json` - JSON file containing generic information about the hub as well as static configuration settings for downstream tools such as validations, visualizations, etc. This file also contains optional cloud settings for hubs that use cloud storage.
2. `tasks.json` - JSON file specifing modeling tasks and model output formats, which may be round-specific.
3. `model-metadata-schema.json` - JSON or YAML file defining format of model metadata files
1. `admin.json` - JSON file containing generic information about the hub as well as static configuration settings for downstream tools such as validations, visualizations, etc. This file also contains optional cloud settings for hubs that use cloud storage. [More details found below](hub_admin_config).
2. `tasks.json` - JSON file specifing modeling tasks and model output formats, which may be round-specific. [More details found below](tasks_metadata).
3. `model-metadata-schema.json` - JSON file defining format of model metadata files. [More information can be found here](model_metadata_schema).

```{caution}
Note: Due to technical issues, we do not currently support json references or yaml metadata files.
Expand Down
3 changes: 2 additions & 1 deletion docs/source/user-guide/model-metadata.md
Expand Up @@ -20,9 +20,10 @@ Note: Model metadata is a required feature of a hub and describes the characteri
## Recommended Standards
Many hubs will use a common set of metadata fields, which we place in the following template schema. Individual hubs may want to add additional metadata fields. These should be selected with the goal of specifying fields that are likely to remain stable across rounds, leaving model attributes that are likely to change across rounds to be described in round-specific abstracts. For consistency across Hubs, we encourage the use of a common set of metadata fields with standard names and definitions. We describe fields that have been used in past Hubs in a second section below the template file.

(model_metadata_schema)=
## Template metadata schema file

The following is a [*template metadata schema file* from the **Example Complex Forecast Hub**](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/blob/main/hub-config/model-metadata-schema.json), which is itself drawn from the [**FluSight Forecast Hub**](https://github.com/cdcepi/FluSight-forecast-hub/blob/main/model-metadata/README.md). Many Hubs will use the same fields listed in the *template metadata schema file*, but some may need fewer or more. A Hub's *model metadata schema file* should include either the single combined field `model_id`, or both of the fields `team_abbr` and `model_abbr`. Administrators can remove other fields as needed, and can use [additional optional metadata fields from this list](#optional-additional-metadata-fields). We encourage administrators to use the same standard names and definitions to maintain consistency across Hubs.
The following is a [template `model-metadata-schema.json` file from the **Example Complex Forecast Hub**](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/blob/main/hub-config/model-metadata-schema.json), which is itself drawn from the [**FluSight Forecast Hub**](https://github.com/cdcepi/FluSight-forecast-hub/blob/main/model-metadata/README.md). Many Hubs will use the same fields listed in the *template metadata schema file*, but some may need fewer or more. A Hub's *model metadata schema file* should include either the single combined field `model_id`, or both of the fields `team_abbr` and `model_abbr`. Administrators can remove other fields as needed, and can use [additional optional metadata fields from this list](#optional-additional-metadata-fields). We encourage administrators to use the same standard names and definitions to maintain consistency across Hubs.

<script src="../_static/docson/widget.js" data-schema="https://raw.githubusercontent.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub/main/hub-config/model-metadata-schema.json"></script>

Expand Down