[
Table 1: Comparison with the existing models on YouTube2Text. SCN takes advantage of ensemble learning. MTVC is a multi-task model.
['Model', 'B-4', 'C', 'M', 'R', 'Overall ( 16 )']
Empirically, we evaluate our method on Youtube2Text/MSVD (guadarrama2013youtube2text) and MSR-VTT (xu2016msr-vtt). LSTM-E (pan2016jointly) makes use of VGG and C3D as visual feature extractors. In LSTM-E, a jointly embedding component is utilized to bridge the gap between visual information and sentence content. h-RNN is composed of a sentence generator and a paragraph generator. The sentence generator of h-RNN exploits temporal-spatial attention mechanism to focus on key segments during generation. The paragraph generator of h-RNN captures dependency between different time step outputs of the sentence generator and provides the sentence generator with new initial state. aLSTMs integrates LSTM with attention mechanism to capture the salient elements in video. What’s more, aLSTMs projects the visual feature and generated sentence feature into a common space and keeps the consistency of semantics by minimizing the Euclidean distance between two embedded features. SCN utilizes a semantics-related variant of LSTM as decoder and exploits C3D and ResNet as encoder. MTVC shares the same model on video captioning task, video prediction task and entailment generation task. The model performance on each task is benefited from the other two tasks. MTVC also utilizes attention mechanism and ensemble learning. Autoencoder for visual information and visual-semantic jointly embedding for semantic information are exploited as encoder in SibNet. The decoder of SibNet generates captions for videos with soft attention. Compared with the previously best results, BLEU-4, CIDEr, METEOR and ROUGE-L are improved by 13.4%, 11.5%, 5.0% and 5.5% respectively.