End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
Table 1: Video Scene-aware Dialog Dataset on Charades
['[EMPTY]', 'training', 'validation', 'test']
In total, we collected dialogs for 7043 videos from the Charades training set and all of the validation set (1863 videos). Since we did not have scripts for the test set, we split the validation set into 732 and 733 videos and used them as our validation and test sets respectively. The average numbers of words per question and answer are 8 and 10, respectively. We evaluated our proposed system with the dialog data for Charades we collected. We compared the performance between models trained from various combinations of the QA text, visual and audio features. In addition, we tested an efficacy of multimodal-attention mechanism for dialg response generation. For each of the encoder-decoder model types, we selected the model with the lowest perplexity on the expanded development set.