Video-Grounded Dialogues with Pretrained Generation Language Models
Table 2: Evaluation on the AVSD benchmark of baselines and different variants of VGD-GPT2 based on: (1) video features in spatial or temporal (or both) dimension and (2) fine-tuning objective functions: MLM - masked language modeling, MVM: mask visual modeling, and MVT - matching video-text pair.
['[BOLD] Model', '[BOLD] Spatial', '[BOLD] Temporal', '[BOLD] MLM', '[BOLD] MVM', '[BOLD] MVT', '[BOLD] BLEU1', '[BOLD] BLEU2', '[BOLD] BLEU3', '[BOLD] BLEU4', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr']
We compare the proposed VGD-GPT2 model with the following baseline models: (1) Baseline hori2019avsd proposes a novel sequence-to-sequence approach with question-guided LSTM on both video visual and audio temporal features. Dialogue history is encoded by a hierarchical LSTM and the final representation is concatenated with question and video representations as input to decode dialog responses. (2) AVSD Winner sanabria2019cmu extends the previous work with more refined visual features and transfer learning from a video summary task. (3) MTN le-etal-2019-multimodal adopts a transformer-based approach with question-guided attention on visual features formulated as an auto-encoding module. To obtain spatial-only and temporal-only features, we follow a similar approach from jang2017tgif by using average pooling to pool the visual features along the temporal or spatial dimensions. Considering CIDEr as the evaluation measure, learning dependencies in both spatial and temporal dimensions can improve the performance by 0.01 absolute score from spatial-only feature and 0.008 absolute score from temporal-only feature.