Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
Table 5: Released by the AVSD organizers, this table shows the final result of objective evaluation values and human rating by using the DSTC8-AVSD test set.
['[EMPTY]', 'BLEU-1', 'BLEU-2', 'BLEU-3', 'BLEU-4', 'METEOR', 'ROUGE-L', 'CIDEr', 'Human']
This result shows that adding important information from a cross-modal way and adding reasoning step help the model better understand videos and then be able to generate correct answers. Moreover, the results also show that the accuracy of JMAN consistently increases when reasoning step n grows. This advantage may come from the additional cross-modal joint-attended features (Tn and Vn) which bring in more information to the model. Nevertheless, for reasoning steps n beyond 5, the model did not show significant increase on every metrics. The best performance of our model (JMAN(DH, C, S, rgb, flow, n=5) achieves 20.8% improvement over the baseline on CIDEr score for DSTC7-AVSD dataset. Therefore, we submitted this best model to DSTC8-AVSD track. Our submitted system outperforms the released baseline model for both subjective and objective evaluation metrics.