Room for improvement in automatic image description: an error analysis
Table 3: Error categories and the BLEU-4 and Meteor scores after correcting the errors. Δ indicates improvement in the scores between the modified descriptions and the original descriptions.
['Type Baseline', 'BLEU 17.8', 'Δ —-', 'Meteor 17.2', 'Δ —-']
For example, after only correcting the colors of clothing, we find a one-point improvement for the BLEU score with respect to the original model. We see two other perspectives to build on the observations from this paper. Automated error analysis: The choices are carefully manipulated, so that each task evaluates the model’s performance in one area (e.g. recognizing scenes). Our taxonomy of errors could be used to extend the range of available tasks, for example with a task to evaluate the use of color terms; Extending existing models: We expect that our work will encourage researchers in vision & language to investigate this possibility. More generally, we hope that our taxonomy of error types will help others to go beyond similarity-based metrics, and to look at their model’s output through a qualitative lens.