Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge
Table 4: Result summary of different model variants of pointer network generator. The results are tested on the test data of AVSD benchmark with test data from the DSTC7.
['[BOLD] Pointer Source Sequence', '[BOLD] BLEU1', '[BOLD] BLEU2', '[BOLD] BLEU3', '[BOLD] BLEU4', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr']
We evaluated our models with different variants of pointer networks by allowing the models pointing to tokens of different combinations of the text input sequences. In these experiments, we choose the video-and-test setting and only use visual features extracted from the pretrained ResNext101. First, most of the MTN models with our proposed pointer network shows improvement over one that only uses a linear transformation to generate tokens. The performance gain is substantial when we allow the models to point to source sequences of video summaries and user queries. However, the performance is slightly affected when we use pointer network to point to tokens in dialogue history because user queries and dialogue history typically contain more useful information than dialogue history to generate system responses. Secondly, when combining different input sequences with multiple pointer networks, the model with the best performance is one that contains pointers to both video summaries and user queries. By extending the pointer network to MTN and adopting a dynamic combination of vocabulary distributions among pointers, we can boost the language generation capability of the models and generate better responses.