Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge
Table 2: Result summary on the test dataset in the AVSD benchmark for DSTC8.
['[BOLD] Visual', '[BOLD] Audio', '[BOLD] BLEU1', '[BOLD] BLEU2', '[BOLD] BLEU3', '[BOLD] BLEU4', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr', '[BOLD] Human']
When only using audio features extracted from VGGish, we note that the performance slightly improves but not significantly as compared to only using visual features. As compared to the original MTN approach [le-etal-2019-multimodal], we noted the difference in performance between models that use either visual or audio features is substantially reduced. We noted similar observations as we compared the difference of performance between models that only use text features and models that use visual features. We expect these performance gains come from using pointer generators which can point to tokens in the source sequences i.e. user queries and video summaries. Since AVSD is formulated as a generation task with evaluation metrics based on similarity between the generated sentences and the ground truth, we could substantially improve the performance by focusing on the language component of the model. We also observed that using a simple ensemble technique could improve the performance, mainly in BLEU-based metrics. In this case, the ensemble strategy acts as a regularization factor on the vocabulary distribution of the output, resulting in more semantically correct output sentences. However, other metrics do not improve or reduce when performing model ensemble. We obtain the human evaluation scores from the organizers for two of our models. Our models achieve human scores of more than 3.6 on a scale of 4 and were ranked top 5 and 6 among all submissions in the AVSD track.