Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
Table 5: Results on WikiText-2 Corpus .
['Method', 'Dev', 'Test']
WikiText-2 Tab. Our baseline, LSTM achieved 1.803 bpc and HCLM model achieved 1.670 bpc. The HCLM with cache outperformed the baseline models and achieved 1.500 bpc. The word level perplexity is 227.30, which is quite high compared to the reported word level baseline result 100.9 with LSTM with ZoneOut and Variational Dropout regularization Merity et al. However, the character-level model is dealing with 76,136 types in training set and 5.87% OOV rate where the word level models only use 33,278 types without OOV in test set. The improvement rate over the HCLM baseline is 10.2% which is much higher than the improvement rate obtained in the PTB experiment.