Neural Machine Translation of Text from Non-Native Speakers
Table 4: BLEU scores on the WMT test set without (clean) and with synthetic grammar errors. The best performing models for each test set are highlighted. When training and test match (highlighted) we generally observe higher results. However, including all clean and noisy data in the training set (mix-all) yields the best results across all datasets, with the highest average BLEU and the lowest variance.
['WMT Training Set', 'En-Es WMT Test Set clean', 'En-Es WMT Test Set drop', 'En-Es WMT Test Set art', 'En-Es WMT Test Set prep', 'En-Es WMT Test Set nn', 'En-Es WMT Test Set sva', 'En-Es WMT Test Set average ± stdev']
The performance of our systems on the synthetic WMT test sets, as measured by detokenized BLEU Papineni et al. When the system is trained only on clean data (first row) and tested on noisy data, it unsurprisingly exhibits degraded performance. We observe significant drops in the range of 1.0–3.6 BLEU.