Detection Evaluation Script in Python

[metro-smiles]

Hi @chengzhag ,

Great work indeed! I had one quick question. Is the evaluation script (for object detection) also available in Python, so that it could be integrated with the testing module directly?

Appreciate your help.

Cheers!

[chengzhag]

Hi,

Thanks for your feedback!

However, since the evaluation kit is provided by Coop, which is implemented in Matlab, and there is no further plan to migrate it to Python, sorry that it can not be integrated into the testing process easily.

You may try to reimplement a Python version of the detection evaluation, which might be not easy.

[metro-smiles]

Appreciate the response.

[metro-smiles]

Hi @chengzhag ,

So I was trying to get the mAP evaluation scores but it seems that the 'co_bdb_3d.mat' and 'co_layout.mat' files are not being generated in the test set. I dug into it a bit and it seems that models/total3d/dataloader.py does not load the 'world_R_inv' key in the 'camera' dict.

Can you please advice me on how to work around/fix this?

[chengzhag]

Hi, @metro-smiles,

In order to evaluate mAP, you need to generate dataset with our modified data processing code as mentioned by README.

[Ivan-VV]

Hi @chengzhag ,
When I run the evaluation script external/cooperative_scene_parsing/evaluation/detections/script_eval_detection.m, I find there is no 'co_bdb_3d.mat' file in the result path gengerated by training.
So how can I get the file to evaluate 3D object detection? Thank u!

[chengzhag]

Hi, @Ivan-VV,

To be more specific, since Total3D has not provided evaluation code, we use code from Coop for evaluation. However, Total3D uses a coordinate frame different from Coop. So we implement a transformation code for different coordinate frames.

However, the transformation is different for each scene and is generated while preparing the dataset with our modified code, and will be needed here when testing (if not exist, the testing process will not generate 'co_bdb_3d.mat').

Thus, you need to prepare the dataset with our modified data processing code in order to do the evaluation.

[Ivan-VV]

Thank you for your reply! @chengzhag
I understand that it is because I didn't run the test code that the file was not generated.