Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features
TABLE I: The Ablation Test of Unimodal Models
['[BOLD] Models', '[BOLD] Arousal (CCC)', '[BOLD] Arousal (MSE)', '[BOLD] Valence (CCC)', '[BOLD] Valence (MSE)']
For visual unimodal model, we investigated the effectiveness of VGG-face and OpenFace features separately in an ablation test. Our results demonstrated that VGG-face features outperformed OpenFace features under the same model architecture. Better performance on both arousal and valence tasks were achieved when the two features are fused. For the audio network, we focused on studying the importance of temporal modeling in utterance. We implemented another LSTM-based network for audio modality. Specifically, we divided each audio file into audio frames of 0.5 second length, and extracted openSMILE features for each single frame. Those features are then fed into a 64 cells LSTM layer followed by the decision layer. The LSTM layer does not benefit the estimation. For text modality, we compared the proposed word frequency statistical approach with models using pretrained word embeddings and LSTM layers in NLP(Natural Language Processing). We implemented the latter approach by using the 100 dimensional GloVe word vectors pretrained on English WikiPedia[pennington2014glove] and a 64 cells LSTM layer in Text(LSTM) model. We compared the performance with text unimodal model using simple opinion lexicon features. Surprisingly, simple lexicon features performed better. This results from the frequently occurring errors as being transcribed using Automatic Speech Recognition Tool in this dataset. The opinion lexicon features, however, mostly ignore these errors by only counting the words appearing in opinion lexicon.