evaluation script gives wrong accuracy

[ovidiunitu]

While testing my solution I noticed this odd behavior. (See the picture below)

As you can see, my generated answer is 'none'.
According to the evaluation metric the correct accuracy should be 30% because there is one answer the same as mine.
I think this is happening because of the processing done before evaluation. In file vqaEval.py line 42, the answer 'none' is replaced with '0' . Because there is not any '0' in ground truth answers, the accuracy is set to 0.00%. If I remove 'none': '0', from manualMap dictionary I get the right accuracy for this question (30%).

If it helps, the id of this question is 411188011, and the name of the picture is COCO_val2014_000000411188.jpg

Can you look more into it? I hope I didn't miss anything.

[AishwaryaAgrawal]

Thanks for bringing up the issue and looking into potential reason! I will look more into it and get back to you.

[guoyang9]

BTW, should we replace all the number words to digits, e.g., 'one on left' to '1 on left'?

Another concern is that some of the answers in the Annotations are just 'a' or 'the', is it appropriate to just delete these ones?

Hope I didn't mistake them.

[baiyuting]

did it get solved?