Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings
Table 5: Results on development set and test sets used in WMT 2018 APE shared task. We show our system’s performance trained by each of the three proposed orderings, and two other models for comparison. correia2019simple is a monotonic model following the sequence-to-sequence architecture and pre-trained on BERT (seq2seq BERT). berard-etal-2017-lig predict a sequence of actions in a left-to-right order.
['[EMPTY]', 'dev 2016 TER ↓', 'dev 2016 BLEU ↑', 'test 2016 TER ↓', 'test 2016 BLEU ↑', 'test 2017 TER ↓', 'test 2017 BLEU ↑', 'test 2018 TER ↓', 'test 2018 BLEU ↑']
We explore three different ways to order the actions provided by minimum edit distance: l2r, shuff and h-ord. For each run, we pick the best checkpoint measured by TER in the development set, and evaluate on 3 test sets. Depending on the dataset chosen, the best performance is achieved by either l2r or h-ord, with small variations between the two. Random shuffling is consistently worse than the alternatives, by a margin of around 0.3 TER. All three alternatives significantly improve the uncorrected MT baseline.