Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
Table 2: Human performance on the DMC task with the MCIC-COCOdataset. Bold denotes the performance ceiling.
['Correct responses', '# instances', 'Accuracy%']
The first row, with accuracy at 67.3%, suggests that this is the level at which the correct answer is obvious (i.e., percentage of “easy” instances). The second row, at 82.8%, indicates that this is the performance ceiling in terms of accuracy that can be expected for the MCIC-COCOdataset; at the same time, it suggests that the difference between 67.3% and 82.8% (i.e., about 15% of instances) is caused by “difficult” instances. Finally, the third row, at 93.1%, indicates that the level of “unanswerable” instances is somewhere in the 10%-15% range (combining the increase from 82.8% to 93.1% and the remaining 6.9% that no one gets right). At accuracy levels of 63.4% (dev) and 60.8% (test), this performance establishes a high-bar for a computer model performance on the DMC task using the MCIC-COCOdataset.