A Purely End-to-end System for Multi-speaker Speech Recognition
Table 4: CER (%) of mixed speech for WSJ.
['Split', 'High E. spk.', 'Low E. spk.', 'Avg.']
The second, third and forth columns indicate CERs of the high energy speaker (High E. spk.), the low energy speaker (Low E. spk.), and the average (Avg.), respectively. The baseline model has very high CERs because it was trained as a single-speaker speech recognizer without permutation-free training, and it can only output one hypothesis for each mixed speech. In this case, the CERs were calculated by duplicating the generated hypothesis and comparing the duplicated hypotheses with the corresponding references. The proposed models, i.e., split-by-VGG and split-by-BLSTM networks, obtained significantly lower CERs than the baseline CERs, the split-by-BLSTM model in particular achieving 14.0% CER. This is an 83.1% relative reduction from the baseline model. The CER was further reduced to 13.7% by retraining the split-by-BLSTM model with the negative KL loss, a 2.1% relative reduction from the network without retraining. This result implies that the proposed negative KL loss provides better separation by actively improving the contrast between the hidden vectors of each speaker. Finally, we profiled the computation time for the permutations based on the decoder network and on CTC. Permutation based on CTC was 16.3 times faster than that based on the decoder network, in terms of the time required to determine the best match permutation given the encoder network’s output in Eq.