Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Before training, check user-provided train_labels to make sure it contains a validation split #214

Open
AllenDowney opened this issue Aug 18, 2022 · 1 comment
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@AllenDowney
Copy link
Contributor

Currently, if the user does not define splits, we generate random splits and check them.

But if the use provides the splits, we don't do any checking. If the user-provided split doesn't have any videos in the validation split, they'll get an error the first time the validation metric is computed. For example:

RuntimeError: Early stopping conditioned on metric `val_macro_f1` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: `train_loss`

It would be nice to generate a more helpful message before training starts.

@ejm714
Copy link
Collaborator

ejm714 commented Aug 18, 2022

This check may be as simple as just verifying that there is at least one train video and at least one val video. We will need to test if this is sufficient if there are species in train that are not in val, and if there is no holdout set specified.

@klwetstone klwetstone added enhancement New feature or request good first issue Good for newcomers labels Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants