Training loss increases while fine-tuning !

[not-hermione]

I have tried to fine-tune the author's model on object detection using the COCO dataset using everything the same as the authors. I have used /configs/swin/mask_rcnn_swin_tiny_patch4_window7_mstrain_480-800_adamw_3x_coco.py and configs/swin/mask_rcnn_swin_tiny_patch4_window7_mstrain_480-800_adamw_1x_coco.py and their corresponding pre-trained checkpoints to initialize the model and fine-tuned for 10 and 5 epochs respectively. The training loss for the first case shows an increasing trend till epoch 6 and then is somewhat stagnant, while for the second case, the training loss increases till epoch 3, then slightly decreased. Why is this happening? Why should the training loss increase?

But the training loss decreases if I randomly initialize the model, which is correct.

with checkpoint - mask_rcnn_swin_tiny_patch4_window7.pth:

with checkpoint - mask_rcnn_swin_tiny_patch4_window7_1x.pth: