SwinV2 for downstream

[sipie800]

Branch

main branch (mmpretrain version)

Describe the bug

I tried to adapt swinv2 in mmpretrain to dino in mmdet.
The config is swinv2 tiny and dino and I use the provided converted weight. Just change the backbone.
Train on coco. The issue is that it won't converge. As soon as the first log output, the cls loss is about 1.3 or higher. And the grad_norm is nan. After a few steps, the grad_norm get a value but the loss go even higher. And after about 5000 steps, all loss go very large number. Finaly the 1st epoch got all 0.0 mAPs.

As long as I rewrite only the backbone cfg into swin tiny(mmdet one or mmpretrain one) or any backbones, the 1st step cls loss is less than 1.1, which means that that's the normal value. And 1st epoch got 0.29 around mAP. So the bug is in mmpretrain swinv2. I've tried many tuned hyp like lr or arch and hear nothing.

Environment

mmdet 3 dev, mmpretrain 1.2

Other information

No response

[sipie800]

Well the thing is fp16 training. If one should get a weird underfiting problem in swinv2, turn off amp.