Approximating probabilistic models as weighted finite automata
Table 2: Comparison of character 5-gram models derived from either the original corpus or a word trigram model. Size of the models is presented in terms of the number of character n-grams, the numbers of states and transitions in the automaton representation, and the file size in MB. The two corpus estimated models have the same topology, hence the same size; as do the two word trigram estimated models.
['Source', '[ITALIC] n-grams (x1000)', 'states (x1000)', 'transitions (x1000)', 'MB', 'Estimation', 'bits/char']
We encode the test set as a sequence of characters, without using the symbol table since our models are intended to be open vocabulary. Following typical practice for open-vocabulary settings, we evaluate with bits-per-character. Here we achieve slightly lower bits-per-character than even what we get straight from the corpus, perhaps due to better regularization of the word-based model than with either Witten-Bell or Kneser-Ney on the character n-grams.