Problems regarding transfer learning

[marialinea]

Hello,

and thank you for making your work available, its been very helpful!

I'm trying to use the pretrained model for the inria dataset on a custom dataset and continuing the training process. I'm almost certain that I have preprocessed the images correctly with corresponding config files. The training process runs smoothly, and I experience no errors. The problem arises when I'm testing the model after training, where the model outputs results that is significantly worse than the pretrained model's output on the custom test data .

To make sure that the custom dataset wasn't the problem I tried to resume training on the original inria dataset with the .geojson files provided in the google drive and the pretrained model. The same problem arises, it looks like the model dosen't continue from the downloaded checkpoints. I've printed a variety of weights from selected layers before and after calling the load_checkpoint function in trainer.py to make sure that the weights do in fact load - and they do.

Do you have any idea where something might go wrong? Are there any variables in the config files that needs to be changed in order to enable transfer learning (To my understanding I think not, but asking just to be on the safe side)? I'm totally in the dark as to what goes wrong and where to look

Thanks in andvance!
Maria

[marialinea]

Update: to my disappointment the transfer learning do happen, although the results are in general quite bad. When resuming training on the inria dataset with the pretrained model for 1 epoch the inference on a unseen test image is noticeably worse and more imprecise corner detection. At the moment I have no idea what causes this behaviour.

If you have some insight on the problem it is much appreciated.
Cheers!

[patriksabol]

Hello @marialinea

I have used pretrained model on the mapping challenge dataset to train my own dataset, however I experienced fast overfitting (after 100k samples) - but I think because I turned affine augmentation off. Nevertheless, I managed to fine-tuned model on my own dataset (better when I trained on my own data from scratch - using pretrained weights on ImageNet, I think). Are you sure, you used 'init_run_name' option with the path to pretrained run when starting training? Can you provide loss curves from tensorboard?

Cheers
Patrik