your results is wrong.

[KimNaser]

I implemented your code using this CMU Panoptic. I got 2.66 as you obtained. The issue is in your test.py code. You should correct it and get MAE around 6.68.
I think if you will report this result. It will affect your first work (6d rotation representation for unconstrained head pose estimation).

I hope that I am wrong and you are correct.

Try to change the batch size =1 and see your results.