Neural Machine Translation of Text from Non-Native Speakers
Table 5: BLEU scores on the JFLEG-es dev and test datasets. Our proposed mix-all model is comparable to the clean model on manually corrected input (cor[0–3]). On noisy input (No corr.) the mix-all outperforms the clean model (25.6>24.2). Preprocessing the noisy input with a GEC model (Auto corr.) further improves results (26.1).
['Training', 'Manual correction cor0', 'Manual correction cor1', 'Manual correction cor2', 'Manual correction cor3', 'Manual correction avg.', 'No corr.', 'Auto corr.']
Our mix-all model generally achieves comparable results when presented with each of the four reference corrections of the test set (corX columns). However, when we use the noisy source sentence as input (No corr column) our mix-all model obtains 1.4 BLEU improvements over the clean model. The difference between the performance of the models when presented with clean and noisy input is another indicator for robustness. On the JFLEG-es test set, the noisy source results in a −3.1 BLEU point drop for the clean model, while the drop for our mix-all model is smaller, at −1.7 BLEU points. We used the publicly available JFLEG outputs of the (almost) state-of-the-art model of Junczys-Dowmunt and Grundkiewicz This experiment envisions a pipeline where the noisy source is first automatically corrected and then translated. As expected, this helps the clean model (by +1.1 BLEU), but our mix-all training helps even further (by another +0.8 BLEU). Interestingly, the automatic GEC system only helps in the test set, while there are no improvements in the dev set. Naturally, since automatic GEC systems are imperfect, the performance of this pipeline still lags behind translating on clean data.