End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
TABLE III: Performance of the BRNN framework implemented with different acoustic features. “CH” stands for channel. “Env” stands for testing environment (“C” is clean; “N” is noisy). “Feature” stands for acoustic feature used in the evaluation (“Acc:” accuracy; “Pre:” precision rate; “Rec:” recall rate;“F:” F1-score).
['CH', 'Env', 'Feature', 'Acc', 'Pre', 'Rec', 'F']
For the ideal channels under clean audio recordings, using a feature representation learnt from Mel-filterbank is slightly better than using a representation learnt from the spectrogram. Both of these feature representations lead to significantly better performance than the system trained with hand-crafted features. Learning flexible feature representations from the raw data lead to better performance than using hand-crafted features, as they are not constrained by pre-defined rules or assumptions. For the practical channels under noisy audio recordings, the model trained with hand-crafted features achieve the worse performance. The feature representation learnt from the Mel-filterbank is able to significantly outperforms the representation learnt from the spectrogram by a large margin (10.5%). The feature representation learnt from the spectrogram is more sensitive to acoustic noise.