Visual Story Post-Editing
Table 3: Average evaluation scores for AREL stories, using the human-edited stories as references. All the automatic evaluation metrics generate lower scores when human judges give a higher rating.
['[EMPTY]', 'Reference: AREL Stories  [BOLD] Edited by Human  [BOLD] BLEU4', 'Reference: AREL Stories  [BOLD] Edited by Human  [BOLD] METEOR', 'Reference: AREL Stories  [BOLD] Edited by Human  [BOLD] ROUGE', 'Reference: AREL Stories  [BOLD] Edited by Human  [BOLD] Skip-Thoughts', '[BOLD] Human Rating']
Automatic evaluation scores do not reflect the quality improvements. APE for MT has been using automatic metrics, such as BLEU, to benchmark progress Libovickỳ et al. However, classic automatic evaluation metrics fail to capture the signal in human judgments for the proposed visual story post-editing task.