Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems
Table 5: Evaluation results on the test set for MTN w/o QAE models in which maximum history length is range from 1 to 3 or 10 (i.e. all dialogue turns possible). We also experiments when all the source sequences are concatenated into one and the decoder only has one attention block on the concatenated sequence. The auto-encoder components are also removed. Best result in each metric is highlighted in bold.
['[BOLD] Max. HisLen', '[BOLD] Concat. Source Sequence?', '[BOLD] BLEU4', '[BOLD] ROUGE-L', '[BOLD] CIDEr']
We experimented our models with text-only input e.g. no video audio or visual features and hence, no auto-encoder layers involved (MTN w/o QAE). We tested cases where the maximum dialogue history length Lmaxhis is limited to 1, 2, or 3 turns only. For each case, we also tried to concatenate all the source sequences, including dialogue history, video caption, and query, into a single sequence and use only one multi-head attention block on this concatenated sequence in each decoding layer (Similar to a Transformer decoder). The results show that concatenating the sequences into one affects the quality of the generated responses significantly. When the input sequences are separated and attended differently by different attention modules, the results improve. This could be explained as different sequences contain different signals to generate responses e.g. dialogue history contains information of references or ellipses in the user queries, user queries include direct signals for feature attention in input videos. Another observation is using all possible dialogue turns in the dialogue history i.e. Lmaxhis=10 achieves the best results. We did not conduct experiments of concatenating source sequences with Lmaxhis=10 due to memory issues with large input sequences.