wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Table 5: Ablations on settings for the masking strategy during pre-training. When masking without overlap, we choose starting time steps with p=0.037 which results in the total number of masked tokens to match the baseline.
['[EMPTY]', 'avg WER', 'std']
For a 15 sec long audio sample, the average mask length is 14.7 time-steps, corresponding to 299ms of audio, with a median of 10 time-steps, and a maximum of about 100 time steps; about 49% of all time-steps in the sample will be masked. Reducing M results in increased prediction accuracy for the self-supervised but the task becomes trivial when spans with length one are masked, leading to poor performance on downstream speech recognition tasks. We also consider other masking strategies: w/o overlap uniform(a,b) samples for each starting index a span length Ms from interval a to b and masks the subsequent Ms time-steps taking care not to overlap with existing spans; poisson(λ) and normal(μ, σ) sample Ms from Poisson and normal distributions.