Significant differences in F1 score

[GisKamal]

Hi,
I'm working on developing an object detection model, aiming to achieve the highest possible F1 score with the meticulously prepared training data I have. However, I've encountered an issue where I receive different F1 scores despite using the exact same parameters and identical training and validation datasets each time. Could you help me understand why there are such significant differences between each test? Please refer to the screenshot in the attachment for more details.

[AdeelH]

How big are the training and validation sets? Are you maybe using a RandomWindowGeoDataset for the validation set?

[GisKamal]

Hello, we used 7 images for training, and 3 for validation with 3024 boxes for training and 73 boxes for validation, indeed, for the F1-Score values we presented, it's using a "random" method, here's a similar test (with exactly the same parameters), but with a "sliding" method for the windows.

[AdeelH]

In that case, the variance makes sense because of the small size of the validation set. One option might be to train for longer, in the hopes that the scores converge.

Re: the sliding windows: it is best to use sliding windows for the validation set so that it is identical across experiments, which makes comparison of metrics more meaningful.

[GisKamal]

Hi,

Thank you for your response. It all makes sense, but I still have a couple of questions:

1. Is it possible to use random windows for training and sliding windows for validation ?
2. What amount of data constitutes a representative validation set for my case ?

[AdeelH]

Is it possible to use random windows for training and sliding windows for validation ?

Yes. See here, for example (assuming you're using RV via CLI by defining a get_config()):
https://github.com/azavea/raster-vision/blob/v0.21.3/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/semantic_segmentation/isprs_potsdam.py#L165-L178

What amount of data constitutes a representative validation set for my case ?

This is a more general ML question (i.e. not RV-specific) and I don't think there is a simple answer to this. You might want to consider things like cross-validation.

[GisKamal]

Thank you Adeel :)