An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization
Table 5: Evaluation results in word accuracy (Acc, %) and CER. The best results in each language have background color. Many identical values in CER are different, but the difference is irrelevant in Chi-square test.
['[EMPTY]', '[BOLD] English Acc', '[BOLD] English CER', '[BOLD] German Acc', '[BOLD] German CER', '[BOLD] Hungarian Acc', '[BOLD] Hungarian CER', '[BOLD] Icelandic Acc', '[BOLD] Icelandic CER', '[BOLD] Swedish Acc', '[BOLD] Swedish CER']
The baseline from \newcitepettersson2014evaluation has very high word accuracy and low CER scores in all five languages. The results in the baseline are obtained using character-level SMT models except for Icelandic, where the combination of a Levenshtein-based method and a dictionary-based method achieved the best results. We use word accuracy and CER to evaluate the predictions. For the historical spelling normalization task, word accuracy is a very important evaluation metric. Moreover, word accuracy is the only evaluation metric in \newcitebollmann2016improve and \newcitebollmann2017learning. However, CER is a good supplement to word accuracy. It is more fine-grained and evaluates the character-level normalizations. In our experiments, we use Levenshtein distance to compute CER. Compared to the other four languages, we get a huge absolute improvement of 12.04% in Hungarian, improving the word accuracy from 80.1% to 92.14%. We get 0.04%, 0.86%, and 2.99% absolute improvement in German, English, and Icelandic, respectively. Our best NMT result in Swedish is still a little lower than the baseline in word accuracy. We attribute the reason to the dataset size, because Swedish has the smallest training set. Hypothesis 1 is that the performance gap between vanilla RNNs and GRUs/LSTMs will not be huge. Att-RNN even performs better than Att-GRU/LSTM in Icelandic. However, Att-RNN is clearly worse than Att-LSTM in German. These results support our Hypothesis 1 well. Hypothesis 3 is that Transformer models are better than soft-attention-based models. It is interesting that English and Hungarian have much more training data compared to the other three languages. This result reveals that Transformer models need more data to exceed RNN-based models.