Modeling Confidence in Sequence-to-Sequence Models
Table 2: Source word confidence for MT. First two columns: Percentage of found errors when selecting 10% and 20% of the data; Final two columns: F-score when using oracle threshold and thresholds optimized on pseudo-labels
['Model', 'Alignment', '10%', '20%', 'Oracle', 'Pseudo-label']
Moreover, we evaluated methods to identify source words with low confidence. In this case the baseline is to map the posterior probabilities to the source sentence using a GIZA Och and Ney Again, we evaluate the approach with the same four scores. As shown in the first two lines, the Giza alignment from source to target performance clearly better than the one from target to source. Therefore, in the remaining experiments, we only evaluate approaches using the source to target alignment. By using the training-test distance approximated by the autoencoder on the encoder states (Enc Auto 256), we directly have an estimate on the source side and so do not need to map target estimates to the source side. In this case, we see improvements over using the posterior probabilities. Again, the pseudo labels perform not as well without using the posterior probabilities. Next, we map the other three measures, decoder hidden states and the combination of encoder or decoder states and output probabilities, using the Giza alignment to the source. Interestingly, this time, solely using the approximation of the training-test similarity is even better than the combination with the output probabilities. The best system is achieved by the autoencoder of the decoder states (Dec Auto 256). We see improvements by 3% and 10% over the posterior probabilities when looking at 10% and 20% of the data. Finally, we tried to use an internal alignment instead of the Giza alignment. Again, we investigated different sizes for the hidden states used to map the posterior probabilities. We can further improve the quality by using a larger hidden layer. Since we need to learn a very complex mapping from source to target hidden states, a larger layer is better. The best performance is achieved using a layer of 8192 hidden units.