Multi-task Learning for Multi-modalEmotion Recognition and Sentiment Analysis
Table 4: Single-task learning (STL) and Multi-task (MTL) learning frameworks for the proposed approach. T: Text, V: Visual, A: Acoustic. Weighted accuracy as a metric is chosen due to unbalanced samples across various emotions and it is also in line with the other existing works Zadeh et al. (2018c).
['[BOLD] Tasks', '[EMPTY]', '[BOLD] F1-score  [BOLD] T', '[BOLD] F1-score  [BOLD] A', '[BOLD] F1-score  [BOLD] V', '[BOLD] F1-score  [BOLD] T+V', '[BOLD] F1-score  [BOLD] T+A', '[BOLD] F1-score  [BOLD] A+V', '[BOLD] F1-score  [BOLD] T+A+V', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] T', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] A', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] V', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] T+V', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] T+A', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] A+V', '[BOLD] Acc (Sentiment) & Weighted-Acc (Emotion)  [BOLD] T+A+V']
In our proposed framework, we aim to leverage multi-modal and contextual information for predicting sentiment and emotion of an utterance simultaneously in a multi-task learning framework. As stated earlier, a video consists of a sequence of utterances and their semantics often have inter-dependencies on each other. We employ three bi-directional Gated Recurrent Unit (bi-GRU) network for capturing the contextual information (i.e., one for each modality). Subsequently, we introduce pair-wise inter-modal attention mechanism (i.e. visual-text, text-acoustic and acoustic-visual) to learn the joint-association between the multiple modalities & utterances. The objective is to emphasize on the contributing features by putting more attention to the respective utterance and neighboring utterances. Motivated by the residual skip connection He et al. the outputs of pair-wise attentions along with the representations of individual modalities are concatenated. Finally, the concatenated representation is shared across the two branches of our proposed network- corresponding to two tasks, i.e., sentiment and emotion classification for prediction (one for each task in the multi-task framework). Sentiment classification branch contains a softmax layer for final classification (i.e. positive & negative), whereas for emotion classification we use sigmoid layer. The shared representation will receive gradients of error from both the branches (sentiment & emotion) and accordingly adjust the weights of the models. Thus, the shared representations will not be biased to any particular task, and it will assist the model in achieving generalization for the multiple tasks. As stated earlier, our proposed approach requires at least two modalities to compute bi-modal attention. Hence, we experiment with bi-modal and tri-modal input combinations for the proposed approach i.e. taking text-visual, text-acoustic, acoustic-visual and text-visual-acoustic at a time. For completeness (i.e., uni-modal inputs), we also experiment with a variant of the proposed approach where we apply self-attention on the utterances of each modality separately. The self-attention unit utilizes the contextual information of the utterances (i.e., it receives u×d hidden representations), applies attention and forward it to the output layer for classification. In the single-task framework, we build separate systems for sentiment and emotion analysis, whereas in multi-task framework a joint-model is learned for both of these problems. For sentiment classification, our single-task framework reports an F1-score of 77.67% and accuracy value of 79.8% for the tri-modal inputs. Similarly, we obtain 77.71% F1-score and 60.88% weighted accuracy for emotion classification.