Answering Questions about Data Visualizations using Efficient Bimodal Fusion
Table 4: Results for the DVQA dataset for PReFIL compared to baselines and existing algorithms.
['[EMPTY]', '[BOLD] Test-Familiar Structure', '[BOLD] Test-Familiar Data', '[BOLD] Test-Familiar Reasoning', '[BOLD] Test-Familiar Overall', '[BOLD] Test-Novel Structure', '[BOLD] Test-Novel Data', '[BOLD] Test-Novel Reasoning', '[BOLD] Test-Novel Overall']
DVQA is split into Test-Familiar, which contains bar charts with words that are also encountered in its Train set, and Test-Novel, which contains bar charts with novel words in them. PReFIL surpasses SANDY by over 40% in accuracy when both the baseline SANDY and our PReFIL method have access to a perfect Oracle OCR, which is emulated by providing the correct text-annotations for all the elements in the images. When using Tesseract OCR, we obtain about a 24% improvement overall on both test sets. To demonstrate that PReFIL’s performance scales with access to better OCR, we also test a version that uses an improved OCR pipeline (see Sec. This further improves PReFIL’s performance by about 11% bringing it closer to the results of the oracle OCR version. When OCR is removed entirely, PReFIL still performs about 11% better than SANDY without OCR, but this ablation renders many data and reasoning questions impossible to answer. Because no human accuracy estimate for DVQA existed, we had people answer 5000 randomly selected questions for 5000 images from the DVQA Test-Novel split. The annotators were shown example QA pairs from each of three question types. We perform post processing on the provided answers to rectify minor answer entry errors. First, we found some annotators used decimal points or spelled out numerals (“5.0” or “five” instead of “5”) despite our instructions to only use integers when answers are numbers. Because DVQA contains only integers, we convert all such occurrences to the nearest integer. For word answers, we allow one character typographic error to be discounted. With perfect OCR, PReFIL surpasses the DVQA human accuracy result across question types. Its performance on reasoning questions is almost 10% greater (absolute), and it exceeds them by almost 8% (absolute) for DVQA’s data questions, which require measurement. However, without perfect OCR humans exceed PReFIL, although the better OCR used for PReFIL does lead to significantly better results than PReFIL with improved OCR. This suggests that the underlying core algorithm and reasoning mechanisms in PReFIL work well for DVQA, and the main limiting factor is OCR.