ASSESSING THE TOLERANCE OF NEURAL MACHINE TRANSLATION SYSTEMS AGAINST SPEECH RECOGNITION ERRORS
Table 3: Ranked evaluation of the SLT utterances containing ASR errors in tst2012. (Left) Counts of the winner decisions and the percentage of all of the decisions that were influenced by ASR errors. (Right) Mean TER scores across each sentence in the ranked set. The remainder of winner decisions are made on error-free ASR transcripts.
['Lab', 'SLT Rank by sentence Winner', 'SLT Rank by sentence Count', 'SLT Rank by sentence Percentage', 'TER (avg) ↓ mmt', 'TER (avg) ↓ neural']
Are there ASR error conditions in which PBMT remains a better solution than NMT, and if so, what are the properties of these utterances that makes them difficult for NMT? We take a closer look at the sentence-level translation scores by ranking the performance of each MT system on the utterances where ASR errors exist, in order to understand how each MT system handles noisy input. For each utterance, we rank the systems based on their the sentence-level TER scores computed on their translation outputs over each ASR hypothesis. We also mark ties, in which both systems yield the same TER score. The neural and mmt scores are tied on over 20% of the utterances. For the better performing ASR systems (e.g. NICT, KIT), we observe a slightly higher proportion of utterances with better NMT translations and a reduced number of ties. we report the average TER scores within each ranking partition of the data. For example, for the utterances that are translated better by mmt, we observe that the average TER scores for neural have an absolute average improvement of 10% in TER over mmt. The converse is also true, suggesting that there is a subset of utterances that mmt translates better than neural.