From Characters to Words to in Between: Do We Capture Morphology?
Table 7: Perplexity results on the Czech development data, varying training data size. Perplexity using ~1M tokens annotated data is 28.83.
['#tokens', 'word', 'char trigram', 'char']
However, we can obtain much more unannotated than annotated data, and we might guess that the character-level models would outperform those based on morphological analyses if trained on larger data. To test this, we ran experiments that varied the training data size on three representation models: word, character-trigram bi-LSTM, and character CNN. Since we want to see how much training data is needed to reach perplexity obtained using annotated data, we use the same output vocabulary derived from the original training. While this makes it possible to compare perplexities across models, it is unfavorable to the models trained on larger data, which may focus on other words. This is a limitation of our experimental setup, but does allow us to draw some tentative conclusions.