Towards VQA Models That Can Read
Table 2: Evaluation on TextVQA. (Left) Accuracies for various heuristics baselines, which show that using OCR can help in achieving a good accuracy on TextVQA. LA+OCR UB refers to maximum accuracy achievable by models using LoRRA with our OCR tokens. (Right) Accuracies of our trained baselines and ablations in comparison with our model LoRRA. I denotes usage of image features, Q question features, O OCR tokens’ features, and C copy mechanism. LA and SA refer to use of large and short vocabulary, respectively. Models with LoRRA outperform VQA SoTA (Pythia, BAN) and other baselines.
['[BOLD] Model', '[BOLD] Accuracy(%)  [BOLD] Val', '[BOLD] Accuracy(%)  [BOLD] Test']
Results. Tab. Despite collecting open-ended answers from annotators, we find that human accuracy is 85.01%, consistent with that on VQA 2.0 While the OCR system we used is not perfect, the upper-bound on the validation set that one can achieve by correctly predicting the answer using these OCR tokens is 37.12%. This is higher than our best model, suggesting room for improvement to reason about the OCR tokens. LA UB is quite high as they contain most commonly occurring questions. This accuracy on VQA 2.0 validation set with 3129 most common answers is 88.9% which suggests uniqueness of answers in TextVQA and limits of a fixed answer space. The difference between LoRRA and LA+OCR UB of 41% represents the room for improvement in modelling with current OCR tokens and LA. Majority answer (“yes”) gets only 4.48% on test set. Random baselines, even the weighted one, are rarely correct. Random OCR token selection and maximum occurring OCR token selection (OCR Max) yields better accuracies compared to other heuristics baselines. Question only (Q) and Image only (I) baseline get 8.09% and 6.29% validation accuracies, respectively, which shows that the dataset does not have significant biases w.r.t. images and questions. I+Q models This demonstrates the inability of current VQA models to read and reason about text in images. A jump in accuracy to 18.35% is observed by feeding OCR tokens (Pythia+O) into the model; this supports the hypothesis that OCR tokens do help in predicting correct answers.