End-to-end Whispered Speech Recognitionwith Frequency-weighted Approaches and Layer-wise Transfer Learning
Table 2: PERs(%) on wTIMIT with only a small normal speech set for training. Section (I) for baselines, Section (II) with the proposed frequency-weighted SpecAugment (FreqSpecAug) with a uniform (UNI), linearly (LIN) and geometrically (GEO) decreasing distribution, Section (III) with frequency-divided CNN extractor (FreqCNN) applied in addition.
['[BOLD] Method', '[BOLD] Method', '[BOLD] (A) Normal  [BOLD] Dev', '[BOLD] (A) Normal  [BOLD] Test', '[BOLD] (B) Whispered  [BOLD] Dev', '[BOLD] (B) Whispered  [BOLD] Test']
Now, we considered the case when only limited normal speech (i.e. the 26 hours of normal speech from wTIMIT) was available for training E2E ASR, but with the several approaches proposed here, to verify these approaches were useful for whispered speech regardless of the training data. The light model same as that used in Sec. The fact that the model performance degraded seriously for whispered speech (columns(B) v.s. (A)) and E2E performed better on normal speech yet worse on whispered (rows(b) v.s. (a)) aligned with the mismatch between whispered and normal, and the assumption that E2E ASR was prone to overfit its training data [Nguyen20-improvesq2sq]. Frequency-weighted SpecAugment To find out the extra robustness achievable by the proposed frequency-weighted SpecAugment, we let the lower end f0 of the mask in SpecAugment to be sampled from a uniform (UNI), linearly (LIN) or geometrically (GEO) decreasing distribution as described in Sec. We see with GEO proposed, a relative 18.8% PER reduction with respect to the baseline E2E (rows(e) v.s. (b)) and a 10.4% relative improvement compared to the original SpecAugment or UNI (rows(e) v.s. (c)) was achieved. This implied with the lower frequencies emphasized in SpecAugment, or letting E2E ASR learn less from lower frequencies, the performance on whispered speech was improved. Obviously in this way we forced the model to distill more details from higher frequencies where whispered speech is more similar to normal speech. Frequency-divided CNN Extractor The results in (rows(f)(g)) show that the frequency-divided CNN made further PER reduction on whispered speech in addition (rows(g) v.s. (e) and (f) v.s. (c)). This verified extracting less lower frequency information with fewer filters while more fine structures or high frequency information with more filters did help, although the accuracy for normal speech was inevitably degraded. The overall relative improvement achieved by the frequency-weighted SpecAugment plus frequency-divided CNN extractor was 19.8% (rows(g) v.s. (b)), which was the setting for the experiments reported below.