Tabula nearly rasa: Probing the linguistic knowledge of character-level neural language models trained on unsegmented text
Table 5: Results on MSR Sentence Completion. For our models (top), we show accuracies for Wikipedia (left) and in-domain (right) training. We compare with language models from prior work (left): Kneser-Ney 5-gram model Mikolov (2012), Word RNN Zweig et al. (2012), Word LSTM and LdTreeLSTM Zhang et al. (2016). We further report models incorporating distributional encodings of semantics (right): Skipgram(+RNNs) from DBLP:journals/corr/abs-1301-3781, the PMI-based model of Woods (2016), and the Context-Embedding-based approach of Melamud et al. (2016).
['[ITALIC] Our models (wiki/in-domain) LSTM', '[ITALIC] Our models (wiki/in-domain) 34.1/59.0']
We confirm the importance of in-domain training, as the models trained on Wikipedia perform poorly (but still above chance level, which is at 20%). With in-domain training, the LSTM CNLM outperforms many earlier word-level neural models, and is only slightly below our WordNLM. The RNN is not successful even when trained in-domain, contrasting with the word-based vanilla RNN from the literature, whose performance, while still below LSTMs, is much stronger. Once more, this suggests that capturing word-level generalizations with a word-lexicon-less character model requires the long-span processing abilities of an LSTM.