Character-Aware Neural Language Models
Table 4: Test set perplexities for Data-s. First two rows are from Botha2014b Botha2014b (except on Arabic where we trained our own KN-4 model) while the last six are from this paper. KN-4 is a Kneser-Ney 4-gram language model, and MLBL is the best performing morphological logbilinear model from Botha Botha2014b. Small/Large refer to model size (see Table 2), and Word/Morph/Char are models with words/morphemes/characters as inputs respectively.
['[EMPTY]', '[EMPTY]', 'Data-s Cs', 'Data-s De', 'Data-s Es', 'Data-s Fr', 'Data-s Ru', 'Data-s Ar']
The model’s performance on the English PTB is informative to the extent that it facilitates comparison against the large body of existing work. However, English is relatively simple from a morphological standpoint, and thus our next set of results (and arguably the main contribution of this paper) is focused on languages with richer morphology The character models also outperform their morphological counterparts (both MLBL and LSTM architectures), although improvements over the morphological LSTMs are more measured. Note that the morpheme models have strictly more parameters than the word models because word embeddings are used as part of the input.