Scene graph baseline for GQA

[yjimmyy]

Hello,

Is there a way to run the scene graph baseline reported in the paper or are there any available details on how to implement it?

[dorarad]

Hi,
Thanks a lot for the interest in the dataset!

what do you mean by scene graph baseline? the baseline reported in the supplementary?
I currently don't provide it -- will add it after NeurIPS. But the implementation is very simple and straight forward: instead of using MAC over the set of object features form the image - I embed each node in the graph based on its symbol (similarly to how question or text is treated). For each node I have a vector:
Concat(Embedding(ObjectName),
Avg(Embedding(Attribute)) over all attributes,
Avg(Concat(Linear(Embedding(Relation),Embedding(RelationTarget))) over all relations)

Where relation target is the object name of the other node participating in this relation. I may release it sooner. And then I get a set of all objects in the image and run standard MAC over that (instead of using visual features).

*Note also that the accuracy there is when testing on the ground-truth scene graphs whereas all other baselines work over the images directly, so the scores should really be compared directly to each other as obviously the direct image task is more difficult. I included this experiment to show a simple ~upper bound of "how well would we do if vision was perfect", and then an ideal model should be in principle achieve 100% in that specific setting.

[ronsoohyeong]

Hi,
After encoding each node as vector as above, did you run CNN-based stem function before the mac network?

[dorarad]

Nope, the CNN stem part was for the original MAC version that worked over the older grid features (by extracting from resnet)