Interpretable Multimodal Routing for Human Multimodal Language
Table 1: Left: CMU-MOSEI sentiment prediction. Right: IEMOCAP emotion recognition. Multimodal Routing∗ denotes our method without iterative routing. Our results are better or close to the state-of-the-art Tsai et al. (2019a). We make our results bold if it is SOTA or close to SOTA (<1%).
['Models', 'CMU-MOSEI Sentiment -', 'CMU-MOSEI Sentiment -', 'CMU-MOSEI Sentiment -', 'IEMOCAP Emotion Happy', 'IEMOCAP Emotion Happy', 'IEMOCAP Emotion Sad', 'IEMOCAP Emotion Sad', 'IEMOCAP Emotion Angry', 'IEMOCAP Emotion Angry', 'IEMOCAP Emotion Neutral', 'IEMOCAP Emotion Neutral']
CMU-MOSEI sentiment. We first compare all the interpretable methods. The improvement suggests the proposed iterative routing can obtain a more robust prediction by dynamically associating the features and the concepts of the model’s predictions. Next, when comparing to the non-interpretable methods, Multimodal Routing outperforms all the other models and performs either closely with or better than SOTA method MulT Tsai et al. Therefore, the proposed method can provide better interpretability while not sacrificing performance. IEMOCAP emotion. We see a performance drop in the emotion “happy”, but our model outperforms the SOTA method for the emotion “angry”.