Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
Table 1: Objective evaluation results on the test set provided by the organizers in DSTC7-AVSD challenge (6 groundtruth available).
['[BOLD] Models', '[BOLD] BLEU-1', '[BOLD] BLEU-2', '[BOLD] BLEU-3', '[BOLD] BLEU-4', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr']
Text only. In detail, our model improves BLEU-4 by 0.069 and CIDEr by 0.185. During human evaluation, the evaluators are asked to rate even the groundtruth references, which are scored 4.000, our model for this task scores 3.934, which is the highest Human rating among all DSTC8 submissions. In the human rating perspective, the results of our model are very close to human dialogue. Text + video. This task uses full information of the track: caption, summary, dialogue history, and video. In particular, our model improved BLEU4 by 0.056, and CIDEr by 0.216. Compared to the text-only task, our models achieve better results, which indicates that our method for video understanding is effective. We adopt multi-task learning as we described before. Video-Audio sequence modeling task improves the score of BELU-4 by 0.011 and CIDEr by 0.026. Text + video w/o caption/summary. This setting is most similar to the actual scene-aware dialogue: we only have video-audio information and dialogue history. Therefore, this task is more challenging. We outperform the DSTC7-AVSD Team 9 who got the highest performance in this task by a large margin. In this task, we also try to use the multi-task learning method but gets lower performance on almost all metrics. We will discuss this phenomenon in the next section.