Lite Audio-Visual Speech Enhancement
Table 1: PESQ and STOI scores of the LAVSE(AE) system and the baselines.
['[EMPTY]', '[BOLD] PESQ', '[BOLD] STOI']
LAVSE versus Audio-only SE First, we intend to compare the performance of the proposed LAVSE with EncoderAE (LAVSE(AE)) face images, and (3) AVSE(lip)—–raw lip images. The results indicate that Audio-only, AVSE(lip), and LAVSE(AE) can notably improve the speech quality (in terms of PESQ) and intelligibility (in terms of STOI). However, AVSE(VGGface) and AVSE(face) cannot yield further improvements compared to the Audio-only system. This may be because the entire face image may contain redundant information that does not directly benefit the SE task. It is also noted that LAVSE(AE) outperforms Audio-only and AVSE(lip) in terms of both PESQ and STOI, suggesting the positive effect of incorporating compressed visual lip information into the SE system. In the following discussion, we present only the results obtained using lip images as visual data. Finally, by combining EncoderAE and Qualatent, the overall compression ratio, RComp is 48 (=RAE×RQua), confirming that the size of the visual input can be significantly reduced using the LAVSE(AE+EOFP) system. From Fig.