Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Table 1: Results on the MSR-VTT dataset.
['Models', 'BLEU-4', 'METEOR', 'ROUGE-L', 'CIDEr']
et al. Ramanishka et al. Then we compare with the state-of-the-art methods on the MSR-VTT dataset: et al. Our HACA model significantly outperforms all the previous methods and achieved the new state of the art on BLEU-4, METEOR, and ROUGE-L scores. Especially, we improve the BLEU-4 score from 41.4 to 43.1. The CIDEr score is the second best and only lower than that of CIDEnt-RL which directly optimizes the CIDEr score during training with reinforcement learning. Note that all the results of our HACA method reported here are obtained by supervised learning only. ATT(v) is a generic attention-based encoder-decoder model that specifically attends to the visual features only. CM-ATT is a cross-modal attentive model, which contains one individual encoder for each input modality and employs a cross-modal attention module to fuse the contexts of different modalities. CM-ATT(va) denotes the CM-ATT model consisting of visual attention and audio attention, while CM-ATT(vad) has an additional decoder attention. Comparing between ATT(v) and CM-ATT(va), we observe a substantial improvement by exploiting the deep audio features and adding cross-modal attention. The results of CM-ATT(vad) further demonstrates that decoder attention was beneficial for video captioning. Note that to test the strength of the aligned attentive decoders, we provide the results of HACA(w/o align) model, which shares almost same architecture with the HACA model, except that it only has one decoder to receive both the global and local contexts. Apparently, our HACA model obtains superior performance, which therefore proves the effectiveness of the context alignment mechanism.