End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
TABLE IV: Performance for unimodal SAD systems and the bimodal SAD system. ‘CH” stands for channel. “Env” stands for testing environment (“C” is clean; “N” is noisy). “Modality” stands for modality used by the approach (“Acc:” accuracy; “Pre:” precision rate; “Rec:” recall rate;“F:” F1-score).
['CH', 'Env', 'Modality', 'Acc', 'Pre', 'Rec', 'F']
For comparison, we also include the proposed BRNN approach. For the ideal channels under clean audio recordings, the bimodal system can outperform the unimodal systems, showing the benefits of using audiovisual features. The result from the audio-based system is 35.6% (absolute) better than the result from the video-based systems. In spite of the lower performance of the visual-based system, the addition of orofacial features lead to clear improvements in the BRNN system. For noisy channels under noisy audio recordings, the bimodal system still achieves the best performance, outperforming the unimodal systems where the differences are statistically significant. The audio-based system achieves better results than the visual-based system. The performance for the visual-based system using noisy audio recordings is higher than the results obtained with clean audio recordings. If we include only read sentences recorded in both noisy and clear recordings, the performance of the visual-based system trained with the ideal channels under clean audio recordings is 69.2%.