Cannot reproduce rtmdet FPS

[mmeendez8]

Describe the bug
I am not able to reproduce the inference speed of rtmdet network.

Reproduction

I have created a simple colab that can help to reproduce my findings. It is a very simple way of running the benchmarking script with different mmdet configs.

I have collected all code in this repo and this is the Colab notebook.

As you can see above rtmdet fps results show that is slower than DETR, Retinanet or FasterRCNN.