Generating Video Descriptions with Topic Guidance
Table 5. Captioning performance with or without using speech modality for topic prediction. The acronyms B, M, R, C denote BLEU@4, METEOR, ROUGE-L and CIDEr respectively.
['[EMPTY]', 'B', 'M', 'R', 'C']
To explicitly explore the usefulness of speech modality in video captioning, we use two kinds of predicted topics which are obtained by the fusion with or without the predictions from speech modality. We can see that the captioning performance achieves large gain in all metrics although the speech modality gets only 1.04% absolute prediction accuracy improvement as mentioned above. It mainly results from the similar topic probability distributions using the shared topic mining model for speech modality. So the speech modality is quite useful to generate topic proposals and thus boost the captioning performance.