End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
TABLE I: Performance of the SAD systems for the ideal channels (close-talking microphone, HD camera). “Env” stands for testing environment (“C” is clean; “N” is noisy). “Modality” stands for modality used by the approach (“A” is A-SAD, “AV” is AV-SAD). “Approach” stands for corresponding framework (“Acc:” accuracy; “Pre:” precision rate; “Rec:” recall rate;“F:” F1-score).
['Env', 'Modality', 'Approach', 'Acc', 'Pre', 'Rec', 'F']
The proposed approach can outperform the baseline approaches (“Ryant-2013” by 0.6%; “Tao-2017” by 4.5%; “Ariav-2018” by 0.5%). The differences between our framework and “Tao-2017” are statistically significant (p-value > 0.05). “Tao-2017” used hand-crafted features. Our end-to-end BRNN system achieves better performance, which demonstrate the benefits of learning the features from the raw data. The proposed approach perform slightly better than the baselines ‘Ryant-2013” and “Ariav-2018”, although the differences are not statistically significant. The proposed approach can outperform the baselines (“Ryant-2013” by 0.7%; “Tao-2017” by 2.3%; “Ariav-2018” by 1.0%). The differences are statistically significant (p-value<0.05) when our approach is compared with “Tao-2017” and “Ariav-2018”. The classification improvement over the “Ariav-2018” system demonstrates that the BRNN framework combine better the modalities than the autoencorder framework, especially in the presence of noise. The proposed BRNN structure jointly learns how to extract the features and fuse the modalities, improving the temporal modeling of the system. This unintuitive result is due to two reasons. First, the signal-to-noise ratios (SNRs) under noisy and clean conditions are very similar for the ideal channels since the microphone is close to the subject’s mouth and far from the audio speaker playing the noise. Second, we only have read speech in the noisy section. In addition to read speech, the clean section also has spontaneous speech, which is a more difficult task for SAD.