Unbabel’s Participationin the WMT19 Translation Quality Estimation Shared Task
Table 2: Word and sentence-level results for En-De and En-Ru on the validation set in terms of F1-MULT and Pearson’s r correlation. (*) Lines with an asterisk use Powell’s method for word level ensembling and ℓ2-regularized regression for sentence level. As the weights are tuned on the dev set, their numbers can not be directly compared to the other models
['Pair', 'System', 'Target  [ITALIC] F1', 'Source  [ITALIC] F1', 'Pearson']
We tried the following strategies for ensembling: For En-De, we created a word-level ensembled system with Powell’s method, by combining one instance of the APE-BERT system, another instance of the Pseudo-APE-QE system, 10 runs of the PredEst-XLM model (trained jointly for all subtasks), 6 runs of the same model without pre-fine-tuning, 5 runs of the PredEst-BERT model (trained jointly for all subtasks), and 5 runs of the PredEst-Trans model (trained jointly for MT and sentence subtasks, but not for predicting source tags). For comparison, we report also the performance of a stacked linear ensembled word-level system. For the sentence-level ensemble, we learned system weights by fitting a linear regressor to the sentence scores produced by all the above models. For En-Ru, we tried two versions of word-level ensembled systems, both using Powell’s method: emsemble 1 combined one instance of the APE-BERT system, 5 runs of the PredEst-XLM model (trained jointly for all subtasks), one instance of the PredEst-BERT model (trained jointly for all subtasks), 5 runs of the NuQE models (trained jointly for all subtasks), and 5 runs of the PredEst-Trans model (trained jointly for MT and sentence subtasks, but not for predicting source tags). emsemble 2 adds to the above predictions from the Pseudo-APE-QE system. In both cases, for sentence-level ensembles, we learned system weights by fitting a linear regressor to the sentence scores produced by all the above models. The PredEst-XLM model, which has been pre-trained with a translation objective, has a small but consistent advantage over both PredEst-BERT and PredEst-Transf. A clear takeaway is that ensembling of different systems can give large gains, even if some of the subsystems are weak individually.