Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
Table 1: State-of-the-art comparison on FVQA dataset.
['[BOLD] Method', '[BOLD] Overall Accuracy  [BOLD] top-1', '[BOLD] Overall Accuracy  [BOLD] top-3']
i.e. Straight to the Facts (STTF) Our model consistently outperforms all the approaches on all the metrics and achieves 3.71% boost on top-1 accuracy and 5.69% boost on top-3 accuracy compared with the state-of-the-art model. The model OB is most relevant to Mucko in that it leverages graph convolutional networks to jointly assess all the entities in the fact graph. However, it introduces the global image features equally to all the entities without selection. By collecting question-oriented visual and semantic information via modality-aware heterogeneous graph convolutional networks, our model gains remarkable improvement.