Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
Table 2: Evaluation scores of commonsense completion task using BLEU, Perplexity, Rouge and Meteor metrics. We use only BLEU-1,2 to evaluate the attribute generation since the average length of the ground truth is less than 4
['[BOLD] Relation  [BOLD] Attribute', '[BOLD] Model', '[BOLD] PPL ↓ Textual Retrieval', '[BOLD] BLEU-1', '[BOLD] BLEU-2 1.85', '[BOLD] BLEU-3 4.48', '[BOLD] BLEU-4 7.90', '[BOLD] METEOR –', '[BOLD] ROUGE-L –', '–', '–']
Natural Language Generation Metrics Compared to the baseline model, our method exhibits a consistent and overall improvement on almost all metrics. Our V2C-Transformer significantly outperforms the LSTM based model in [gao2017video] by 6.3% at BLEU-4 for the intention prediction, surpassing the performances of Textual Retrieval which follows our data construction procedure. It’s worth noting that retrieving commonsense knowledge from Atomic also yields competitive scores with the learning methods, but since this is not learnt, it cannot be applied to novel videos at test time. Because the V2C-Transformer and the LSTM model share a similar video encoder, our performance improvement could be attributed to the use of self-attention mechanisms in the transformer block in decoding phase. This observation is consistent with the conclusion from [bosselut2019comet], and yields further support to the transformer architecture being suited for commonsense inference tasks. Moreover, when compared with the Dense Captioning model with similar transformer architectures and parameters, our model exhibits better evaluation scores, verifying it as a strong baseline model for the V2C task.