Abstract
Table 5: Correlation of BLEU*, METEOR* and STS scores for submissions participating in in TRECVid 2016 VTT task with human assessment.
['[EMPTY]', 'BLEU*', 'METEOR*', 'STS', '[ITALIC] n']
As mentioned previously, in MT the use of automatic metrics is validated by how well a given metric correlates with human assessment. Since now have human assessment of captions for systems participating in TRECVid 2016 VTT, we can therefore examine how well metric scores used to produce the official results in last year’s benchmark correlate with human assessment. *, METEOR * and STS, as applied to captions in last year’s VTT task, correlate with human assessment. * scores correspond reasonably well to human evaluation, with the main disagreement taking place for the top two systems, Sheffield and MediaMill, the former outperforming the latter according to human evaluation. In contrast BLEU * scores incorrectly report a higher score for MediaMill, although only marginally so. * scores and human assessment compared to BLEU*, where the system scored highest according to human judges and significantly outperforming all other systems, Sheffield, is ranked worst according to this metric. This disagreement in fact results in a somewhat disconcerting negative correlation of METEOR