Multimodal Matching Transformer for Live Commenting
Table 2: The performance comparison on the live commenting dataset (Recall@k, MRR: higher is better; MR: lower is better). Our matching transformer significantly outperforms the baselines in terms of all metrics. Meanwhile, our model achieves better performance than baselines by using the same two modalities.
['[BOLD] Model', '[BOLD] Text', '[BOLD] Vision', '[BOLD] Audio', '[BOLD] Recall@1', '[BOLD] Recall@5', '[BOLD] Recall@10', '[BOLD] MR', '[BOLD] MRR']
The baselines (S2S, Fusional RNN and Unified Transformer) only use the text and vision of the videos. Our matching transformer leverages three modalities (text, vision, and audio) and significantly outperforms the baselines in terms of all metrics. Moreover, we also report the results of our model with only one modality (less than baselines) and two modalities (equal to baselines). It shows that the matching transformer achieves comparable performance with the baselines using only one modality. Meanwhile, our model achieves better performance than baselines by using the same two modalities, which verifies the efficiency of the proposed model. Finally, the triple-modality model significantly outperforms the baselines, achieving +5.51 points on Recall@1 , +8.87 points on Recall@5 and +8.46 points on Recall@10.