MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Table 3: Human evaluation on ActivityNet Captions ae-test set w.r.t. relevance and coherence. Top: MART vs. vanilla transformer (VTransformer). Bottom: MART vs. Transformer-XL.
['[EMPTY]', 'MART wins (%)', 'VTransformer wins (%)', 'Delta']
In addition to the automatic metrics, we also run human evaluation on Amazon Mechanical Turk (AMT) with 200 randomly sampled videos from ActivityNet Captions ae-test split, where each video was judged by three different AMT workers. , where we compare two models at a time. AMT workers are instructed to choose which caption is better or the two captions are not distinguishable based on relevance and coherence, respectively. The models are anonymized, and the predictions are shuffled. In total, we have 54 workers participated the MART vs. vanilla transformer experiments, 47 workers participated the MART vs. Transformer-XL experiments. With its sentence-level recurrence mechanism, MART is substantially better than the vanilla transformer model for both relevance and coherence. Compared to the strong baseline approach Transformer-XL, MART has similar performance in terms of relevance, but still reasonably better performance in terms of coherence.