An Analysis of Visual Question Answering Algorithms
Table 5: Results for all the VQA models. The normalized accuracy for each question-type is shown here. The models are identical to the ones in Table 3 in main paper. Overall performance is, again, reported using all 5 metrics. Overall (Arithmetic N-MPT) and Overall (Harmonic N-MPT) are averages of the reported sub-scores. Similarly, Arithmetic MPT and Harmonic MPT are averages of sub-scores reported in Table 3 in the main paper. * denotes training without absurd questions.
['[EMPTY]', '[BOLD] YES', '[BOLD] REP', '[BOLD] IMG', '[BOLD] QUES', '[BOLD] Q+I', '[BOLD] *Q+I', '[BOLD] MLP', '[BOLD] MCB', '[BOLD] *MCB', '[BOLD] MCB-A', '[BOLD] NMN', '[BOLD] RAU']
The unnormalized accuracy for each question-type is shown. Overall performance is reported using 5 metrics. Overall (Arithmetic MPT) and Overall (Harmonic MPT) are averages of these sub-scores, providing a clearer picture of performance across question-types than simple accuracy. Overall Arithmetic N-MPT and Harmonic N-MPT normalize across unique answers to better analyze the impact of answer imbalance (see Sec. * denotes training without absurd questions. One of our major aims was to compensate for the fact that algorithms can achieve high scores by simply learning to answer more populated and easier question-types. On TDIUC, MLP surpasses MCB and NMN in terms of simple accuracy, but a closer inspection reveals that MLP’s score is highly determined by performance on categories with a large number of examples, such as ‘absurd’ and ‘object presence.’ Using MPT, we find that both NMN and MCB outperform MLP. This indicates that MLP is prone to overfitting. Similar observations can be made for MCB-A compared to RAU, where RAU outperforms MCB-A using simple accuracy, but scores lower on all the metrics designed to compensate for the skewed answer distribution and bias.