Character-Aware Neural Language Models
Table 7: Perplexity on the Penn Treebank for small/large models trained with/without highway layers.
['[EMPTY]', 'LSTM-Char Small', 'LSTM-Char Large']
One could also have a multilayer perceptron (MLP) over yk to model interactions between the character n-grams picked up by the filters, but we found that this resulted in worse performance. We train a model without any highway layers, and find that performance decreases significantly. As the difference in performance could be due to the decrease in model size, we also train a model that feeds yk (i.e. word representation from the CharCNN) through a one-layer multilayer perceptron (MLP) to use as input into the LSTM. We find that the MLP does poorly, although this could be due to optimization issues.