Tabula nearly rasa: Probing the linguistic knowledge of character-level neural language models trained on unsegmented text
Table 1: Performance of language models. For CNLMs, we report bits-per-character (BPC). For WordNLMs, we report perplexity.
['[EMPTY]', '[ITALIC] LSTM', '[ITALIC] RNN', '[ITALIC] WordNLM']
Recall that we removed whitespace, which is both easy to predict, and aids prediction of other characters. Consequently, the fact that our character-level models are below the state of the art is expected. For example, the best model of \newcitemerity2018analysis achieved 1.23 English BPC on a Wikipedia-derived dataset. On EuroParl data, \newcitecotterell2018all report 0.85 for English, 0.90 for German, and 0.82 for Italian. Still, our English BPC is comparable to that reported by \newciteDBLP:journals/corr/Graves13 for his static character-level LSTM trained on space-delimited Wikipedia data, suggesting that we are achieving reasonable performance. The perplexity of the word-level model might not be comparable to that of highly-optimized state-of-the-art architectures, but it is at the expected level for a well-tuned vanilla LSTM language model. For example, \newciteGulordava:etal:2018 report 51.9 and 44.9 perplexities respectively in English and Italian for their best LSTMs trained on Wikipedia data with same vocabulary size as ours. This was due to resource availability.