Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check that weights don't depend on output_type_id in settings where that would not lead to a valid ensemble #35

Open
elray1 opened this issue Nov 1, 2023 · 1 comment

Comments

@elray1
Copy link
Contributor

elray1 commented Nov 1, 2023

Mainly, I think this is a check we need when the output type is pmf or cdf. In that case, allowing the weights to depend on the output_type_id could lead to an invalid predictive distribution. We could implement this check in hubEnsembles::simple_ensemble, and I think that would be good enough since hubEnsembles::linear_pool calls simple_ensemble.

I initially thought that we should also not allow weights to depend on the sample index if the output type is sample, but I don't think there's necessarily anything wrong with a per-sample weighting, e.g. if the hub or modeler has some extra information about how the different samples are generated and wants to weight them based on that that factor.

@elray1
Copy link
Contributor Author

elray1 commented Nov 3, 2023

We might want to allow users to manually override whether this check is done. For example, if you're careful about your weighting scheme this could be OK. And we might want to be able to do this for a trimmed linear pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant