TextCaps: a Dataset for Image Captioningwith Reading Comprehension
Table 1: Performance of our baselines on our TextCaps dataset. M4C-Captioner significantly benefits from training with OCR inputs, and achieves the highest CIDEr score among all methods, suggesting that it is important to learn to copy text from image when generating a caption. However, we see that there is still a large gap between the current machine performance and human performance on this task, which we hope can be closed by future work.
['#', 'Method', 'Trained on', 'TextCaps validation set metrics B-4', 'TextCaps validation set metrics M', 'TextCaps validation set metrics R', 'TextCaps validation set metrics S', 'TextCaps validation set metrics C', '[EMPTY]']
When trained on the TextCaps dataset (line 2), the BUTD model has higher scores as expected, since there is no longer a domain shift between training and evaluation. AoANet (line 3, 4), which is a stronger captioning model, outperforms BUTD but still cannot handle reading comprehension and largely underperforms M4C-Captioner. For the M4C-Captioner model, there is a large gap (especially in CIDEr scores) between training with and without OCR inputs (line 13 vs. 7). Moreover, “M4C-Captioner w/o copying” (line 8) is worse than the full model (line 13) but better than the more restricted “M4C-Captioner w/o OCRs” (line 7). The results indicate that it is crucial to both encode OCR features and be able to directly copy OCR tokens. We also observe (in line 13 vs. 9-12) that it is important for a model to use spatial, visual, and semantic features of OCR tokens together, especially in the complex combinations of OCR tokens where both spatial relation and semantics play crucial role in finding a connection between words. However, on the test set, we still notice a large gap between the best machine performance (line 17) and the human performance (line 19) on this task. Also, using ground-truth OCRs as inputs (line 18) reduces this gap but still does not close it, suggesting that there is room for future improvement in both better reasoning and better text recognition.