Multi-talker ASR for an unknown number of sources:Joint training of source counting, separation and ASR
Table 3: Recognition performance of the multi-talker ASR systems for varying numbers of speakers given the oracle number of sources. The mixture signals contain full utterances (max subset). (“FE”: front-end, “ASR”: speech recognition part.)
['Experiment', 'Experiment', 'WSJ0-2mix CER', 'WSJ0-2mix WER', 'WSJ0-2mix SDRi', 'WSJ0-3mix CER', 'WSJ0-3mix WER', 'WSJ0-3mix SDRi', 'WSJ0-4mix CER', 'WSJ0-4mix WER', 'WSJ0-4mix SDRi']
We observe similar tendencies as were found in [Heymann2017_BeamnetEndtoendTraining] and [vonNeumann2019_EndtoendTrainingTime]. Fine-tuning the ASR part gives a greater improvement than fine-tuning the TasNet but fine-tuning both jointly gives the best performance on two speakers (compare rows 1 to 3) of 58% relative improvement compared to not fine-tuning. The final 7.5% WER (3) forms a new state-of-the-art result on WSJ0-2mix and is a huge improvement over previous techniques [vonNeumann2019_EndtoendTrainingTime]. Similar to the separation evaluation, the two-speaker TasNet performs better than the two- and three-speaker model (rows 7 to 9). We use the stop-flag OR-PIT due to better counting accuracy on max data. The OR-PIT again performs better than the TasNet on more than two speakers. It generalizes to the unseen larger number of four speakers. Contradicting the results for two speakers, fine-tuning just the ASR part performs best for more than two speakers (rows 5, 8, 12 and 14). These results suggest that it is beneficial to only fine-tune the ASR back-end in some scenarios, especially when the separation task is more challenging, while fine-tuning the front-end contributes to over-fitting to a specific scenario.