Self-supervised Learning of Visual Speech Features with Audiovisual Speech Enhancement
Table 1: Enhancement performance obtained for each setup and loss function. SNR: signal-to-noise-ratio in decibel (dB), PESQ: perceptual evaluation of speech quality, A: Audio-only enhancer, AV: audiovisual enhancer, FC: fully connected, LSTM: Long short-term memory
['[BOLD] Audio Encoder', '[BOLD] Loss', '[BOLD] SNR  [BOLD] A', '[BOLD] SNR  [BOLD] AV', '[BOLD] PESQ  [BOLD] A', '[BOLD] PESQ  [BOLD] AV']
The results show that considerable gains are achieved using an audio-only enhancer (columns labeled A), which was not expected a priori. One reason for this is that although the target mixture for the noisy signal was 0dB SNR, mixtures of 1dB emerged due to short pauses in the target and background speech. This 1dB difference between target and background acoustic speech gives the network a clue for enhancing the target speaker, even without visual cues. That said, the results show that the addition of visual cues still provides improvement in performance for all setups. The results also show that using an LSTM-based audio encoder yields better performance compared to FC-based encoders. This can be attributed to the temporal modeling of LSTMs. Finally, the results show that using the proposed hybrid loss function gives improvements over using MSE for the majority of the setups.