Evaluating historical text normalization systems: How well do they generalize?
Table 2: Tokens normalized correctly (%) for each dataset. Upper half: results on (A)ll tokens reported by Pettersson et al. (2014) for a hybrid model (apply memorization baseline to seen tokens and an edit-distance-based model to unseen tokens) and two SMT models (which align character unigrams and bigrams, respectively). Lower half: results from our experiments, including accuracy reported separately on (S)een and (U)nseen tokens.
['[EMPTY]', 'English A', 'English S', 'English U', 'German A', 'German S', 'German U', 'Hungarian A', 'Hungarian S', 'Hungarian U', 'Icelandic A', 'Icelandic S', 'Icelandic U', 'Swedish A', 'Swedish S', 'Swedish U']
The split into seen/unseen highlights the fact that neither of the neural models does as well on seen items as the baseline; indeed the soft attention model is considerably worse in English and Hungarian, the two largest datasets. The result is that this model actually underperforms the baseline when applied to all tokens, although a hybrid model (baseline for seen, soft attention for unseen) would outperform the baseline. Nevertheless, the hard attention model performs best on unseen tokens in all cases, often by a wide margin, and also yields competitive overall performance.