ASSESSING THE TOLERANCE OF NEURAL MACHINE TRANSLATION SYSTEMS AGAINST SPEECH RECOGNITION ERRORS
Table 1: A comparison of Neural MT versus Phrase-based MT on the SLT evaluation of TED talks (tst2012) from the IWSLT 2013 evaluation campaign [12]. Evaluation results are compared to a gold standard that assumes that no ASR errors occur.
['ASR system', 'WER ↓', 'mmt BLEU ↑', 'mmt TER ↓', 'neural BLEU ↑', 'neural TER ↓']
NMT outperforms our best PBMT system by 4.5 BLEU in the absence of ASR errors (gold) and by approximately 3 BLEU across all ASR hypothesis inputs. Overall, the introduction of ASR errors results in decreases in BLEU by 5.5(±0.8) and 5.4(±0.8) and TER increases of 6.0(±0.9) and 6.2(±0.9) for mmt and neural, respectively.