Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition
Table 4: Comparisons on the test set. LST represents our proposed “Learn Spelling from Teachers” approach. SF means using shallow fusion during decoding.
['Model', 'CER%']
“Seq2Seq” is the plain Seq2Seq without regularization. Compared to “Seq2Seq”, “Seq2Seq+LST” achieves an 18.42% relative reduction in character error rate. We analyze that the assumptions of label smoothing (uniform distribution) and unigram smoothing (unigram frequency) do not match real situations. However, LST, which is a data-driven approach, does not assume prior distributions.