Self-supervised Learning of Visual Speech Features with Audiovisual Speech Enhancement
Table 3: Viseme classification performance obtained using visual embeddings extracted from the video encoder in the enhancement model.
['[BOLD] Viseme', '[BOLD] Recall (%)']
We find that the visual embeddings are discriminative toward visemes, giving an overall unweighted accuracy of 33.5%, where 7.7% is the chance performance. We find that our classifier predicts apparent visemes, such as /Z/, /F/, /P/, and /SH/, relatively accurately compared to predicting visemes articulated more towards the back of the mouth, such as /T/ and /G/. The trends that we observe for viseme prediction performance using visual embeddings are similar to those observed in viseme classification tasks. As a benchmark, we were able to obtain an unweighted accuracy of 49.2% using a separate VGG-M neural network trained from scratch specifically to detect visemes, which suggests that our self-supervised visual features were able to close a large proportion of the performance gap. This demonstrates the efficacy of audiovisual speech enhancement as a self-supervised task for learning strong visual features.