Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for the sample model_output_type in hubEnsembles::linear_pool() #101

Open
nickreich opened this issue Apr 4, 2024 · 3 comments

Comments

@nickreich
Copy link
Contributor

nickreich commented Apr 4, 2024

Add support for the sample model_output_type in hubEnsembles so that hub administrators and/or individual modelers can run linear pool ensembles on model_output in sample format.

Acceptance criteria:

  • sample output_type can be passed to linear_pool() with valid ensemble output returned
  • unit tests in place to check accuracy of calculations
  • documentation of function, including arguments and explanations for any analysis choices that need to be specified by the user
  • warning is thrown if the number of samples available for all prediction units and from all models are not equal
@nickreich nickreich added this to the sample output_type v1.0 milestone Apr 4, 2024
@elray1
Copy link
Contributor

elray1 commented Apr 4, 2024

We should discuss and combine this issue with #27

@elray1
Copy link
Contributor

elray1 commented Apr 4, 2024

r.e. "warning is thrown if the number of samples available for all prediction units and from all models are not equal" -- is that something that is a part of an ensembling operation? feels like a validation that should happen somewhere else?

@nickreich
Copy link
Contributor Author

I think if a user passes model output that has varying numbers of samples per prediction unit (maybe excluding 0s?) there should be a warning thrown. It feels like the kind of thing that could be overlooked but could have weird consequences depending on the implementation.

@nickreich nickreich changed the title add support for building ensembles using sample data in linear_pool() Add support for the sample model_output_type in hubEnsembles::linear_pool() Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

2 participants