Exploring Prediction Uncertainty in Machine Translation Quality Estimation
Table 1: Intrinsic evaluation results. The first three rows in each table correspond to standard GP models, while the remaining rows are Warped GP models with different warping functions. The number after the tanh models shows the number of terms in the warping function (see Equation 2.3). All r scores have p<0.05.
['[BOLD] English-Spanish - 858 instances', '[BOLD] English-Spanish - 858 instances NLL', '[BOLD] English-Spanish - 858 instances NLPD', '[BOLD] English-Spanish - 858 instances MAE', '[BOLD] English-Spanish - 858 instances  [ITALIC] r']
The first two columns shows an interesting finding in terms of model learning: using a warping function drastically decreases both NLL and NLPD. The main reason behind this is that standard GPs distribute probability mass over negative values, while the warped models do not. For the fr-en and en-de datasets, NLL and NLPD follow similar trends. This means that we can trust NLL as a measure of uncertainty for these datasets. However, this is not observed in the en-es dataset. Since this dataset is considerably smaller than the others, we believe this is evidence of overfitting, thus showing that NLL is not a reliable metric for small datasets.