Towards Universal End-to-End Affect Recognition from Multilingual Speech by ConvNets
TABLE III: Results (percentage) on multilingual task for Big Five personality traits analyzed. Mean Absolute Error (MAE), Accuracy (A), Precision (P), Recall (R), and F-score (F1) are shown.
['Method', 'English MAE', 'English A', 'English P', 'English R', 'English F1', 'Mandarin MAE', 'Mandarin A', 'Mandarin P', 'Mandarin R', 'Mandarin F1', 'French MAE', 'French A', 'French P', 'French R', 'French F1']
The fine-tuned multilingual model performs best on the test sets in terms of F-score. For the multilingual model using raw waveforms, we obtained an average F-score over the three languages of 62.4%. Training this same model on each language individually resulted in an average F-score over the languages of 56.7%. Using a spectrogram instead of raw waveforms gives an average F-score of 58.8%. Thus, our multilingual efforts show a relative improvement of 6.3% over the spectrogram approach and 10.1% over the single language approach.