Iterative Refinement for Machine Translation
Table 1: Accuracy of the error detection model f compared to baselines on the concatenation of the WMT test sets from 2008 to 2015. For precision, recall and F1 we consider a positive prediction as labeling a word as a mistake. Baseline fcor labels all words as correct, fwrong labels all words as incorrect, fstat labels a word from yg based on the prior probability estimated on the training data.
['Metric (%)', '[ITALIC] fcor', '[ITALIC] fwrong', '[ITALIC] fstat', '[BOLD] f']
While the model significantly improves over the baselines, the probability of correctly labeling a word as a mistake remains low (62.71%). Also, one should bear in mind that this task cannot be solved with 100% accuracy since a sentence can be correctly in multiple different ways and we only have a single reference translation. In our case, our final refinement objective might be easier than error detection as we do not need to detect all errors. We need to identify some of the locations where a substitution could improve BLEU. At the same time, our strategy should also suggest these substitutions. This is the objective of the model introduced in the next section.