Video Captioning with Guidance of Multimodal Latent Topics
Table 1. Caption performance comparison between Vanilla (no topic guidance) and TGM (using different latent topic guidance) on the MSR-VTT and Youtube2Text datasets. The best results are marked in bold and the second best results with underline.
['model', 'topic guidance', 'MSR-VTT BLEU4', 'MSR-VTT METEOR', 'MSR-VTT ROUGE [ITALIC] l', 'MSR-VTT CIDEr', 'Youtube2Text BLEU4', 'Youtube2Text METEOR', 'Youtube2Text ROUGE [ITALIC] l', 'Youtube2Text CIDEr']
The numbers of textual and multimodal latent topics are optimized on the validation set (50 topics for MSR-VTT and 5 topics for Youtube2Text). We can see that TGMs outperform Vanilla model consistently on all four metrics across MSR-VTT and Youtube2Text datasets, which demonstrates that exploiting topic information is beneficial to generate video descriptions. We can see that the automatically mined topics (both textual and multimodal latent topics), outperform expert-defined topics by a large margin on all four metrics and two datasets, such as over 2% absolute gain on the CIDEr score. The multimodal latent topics further achieve better performance than textual latent topics consistently on multiple metrics across datasets. The improvement is also proved to be significant in the Student’s t-test such as p-value of 0.03 on the BLEU@4 score. These results suggest that the multimodal topic mining approach can discover better underlying topics as guidance for TGM, and such improvement is prominent and robust for different evaluation metrics and datasets. However, the assigned expert-defined topics are very noisy, because the annotations are labelled on the entire videos but the caption generation is only applied on the short segments and the users may also make labelling mistakes. For almost all metrics, the performances of our predicted latent topics are superior to the noisy assigned expert-defined topics on both MSR-VTT and Youtube2Text datasets, which shows that the latent topics learned in an unsupervised approach are competitive or even better than the assigned expert-defined topics with noise.