Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
Table 2: Objective and subjective evaluation results on the test set provided by the organizers in DSTC8-AVSD challenge (6 groundtruth available).
['[BOLD] Models', '[BOLD] BLEU-1', '[BOLD] BLEU-2', '[BOLD] BLEU-3', '[BOLD] BLEU-4', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr', '[BOLD] Human rating']
Text only. In detail, our model improves BLEU-4 by 0.069 and CIDEr by 0.185. During human evaluation, the evaluators are asked to rate even the groundtruth references, which are scored 4.000, our model for this task scores 3.934, which is the highest Human rating among all DSTC8 submissions. In the human rating perspective, the results of our model are very close to human dialogue. Text + video. This task uses full information of the track: caption, summary, dialogue history, and video. In particular, our model improved BLEU4 by 0.056, and CIDEr by 0.216. Compared to the text-only task, our models achieve better results, which indicates that our method for video understanding is effective. We adopt multi-task learning as we described before. Video-Audio sequence modeling task improves the score of BELU-4 by 0.011 and CIDEr by 0.026.