Multimodal Matching Transformer for Live Commenting
Table 3: Effect of different modalities used in the Matching Transformer (Recall@k, MRR: higher is better; MR: lower is better). It shows that more modalities always lead to better performance, which indicates that the proposed model can capture the semantic information of different modalities to help the live commenting task.
['[EMPTY]', '[BOLD] Text', '[BOLD] Vision', '[BOLD] Audio', '[BOLD] Recall@1', '[BOLD] Recall@5', '[BOLD] Recall@10', '[BOLD] MR', '[BOLD] MRR']
We also would like to know how different kinds of modalities contribute to our proposed model. Therefore, we conduct ablation experiments by removing different modalities from our model. Under the single-modality setting, it shows that the model with the modality of text achieves better performance over the other two modalities. Among the possible alternatives of double-modality, the combination of text and vision obtains the best performance. Finally, the model with triple-modalities get the highest scores in terms of all the automatic metrics. Besides, it is observed that more modalities always lead to better performance, which indicates that the proposed model can capture the semantic information of different modalities to help the live commenting task.