End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
TABLE II: Performance of the SAD systems for the practical channels (microphone and camera from the tablet). “Env” stands for testing environment (“C” is clean; “N” is noisy). “Modality” stands for modality used by the approach (“A” is A-SAD, “AV” is AV-SAD). “Approach” stands for corresponding framework (“Acc:” accuracy; “Pre:” precision rate; “Rec:” recall rate;“F:” F1-score).
['Env', 'Modality', 'Approach', 'Acc', 'Pre', 'Rec', 'F']
For clean audio recordings, the proposed approach can significantly outperforms all the baselines (“Ryant-2013” by 0.8%; “Tao-2017” by 4.3%; “Ariav-2018” by 0.8%). For noisy audio recordings, we observe that the performances drop across conditions compared to the results obtained under clean recordings. The microphone of the tablet is closer to the audio speaker playing the noise, so the SNR is lower The proposed approach can maintain a 92.7% F1-score performance, outperforming all the baseline frameworks (“Ryant-2013” by 1.2%; “Tao-2017” by 6.7%; “Ariav-2018” by 1.0%). The differences are statistically significant for all the baselines. This result shows that the proposed end-to-end BRNN framework can extract audiovisual feature representations that are robust against noisy audio recordings.