Character-Aware Neural Language Models
Table 5: Test set perplexities on Data-l. First two rows are from Botha2014b Botha2014b, while the last three rows are from the small LSTM models described in the paper. KN-4 is a Kneser-Ney 4-gram language model, and MLBL is the best performing morphological logbilinear model from Botha2014b Botha2014b. Word/Morph/Char are models with words/morphemes/characters as inputs respectively.
['[EMPTY]', '[EMPTY]', 'Data-l Cs', 'Data-l De', 'Data-l Es', 'Data-l Fr', 'Data-l Ru', 'Data-l En']
The model’s performance on the English PTB is informative to the extent that it facilitates comparison against the large body of existing work. However, English is relatively simple from a morphological standpoint, and thus our next set of results (and arguably the main contribution of this paper) is focused on languages with richer morphology Interestingly we do not observe significant differences going from word to morpheme LSTMs on Spanish, French, and English. The character models again outperform the word/morpheme models. We also observe significant perplexity reductions even on English when V is large. We conclude this section by noting that we used the same architecture for all languages and did not perform any language-specific tuning of hyperparameters.