The Unreasonable Volatility of Neural Machine Translation Models
Table 5: Mean oscillations for SUBNUM variations. In theory the variations should result in zero oscillations for every metric.
['[EMPTY]', 'BLEU', 'METEOR', 'TER', 'LengthRatio']
While sentence-level metrics are not reliable indicators of translation quality, they do capture fluctuations in translations. With the variations we introduce, in theory there should be no fluctuations in the translations. We observe that even though these sentence variations differ by only one number, there are many cases where an insignificant change in the sentence results in unexpectedly large oscillations. Both RNN and Transformer exhibit this behaviour to a certain extent.