Behavior Gated Language Models
Table 2: Language model perplexities on validation and test sets of Penn Treebank.
['[BOLD] Model', '[BOLD] # Params', '[BOLD] Validation', '[BOLD] Test']
Although the proposed model is motivated and targeted towards behavior related datasets, the hypothesis should theoretically extend towards any human generated corpora. To assess this, we also train models on a non-behavior-rich database, the Penn Tree Bank Corpus. We experiment with both the medium and large architectures proposed by Zaremba et al. All language models showed an improvement in perplexity through the addition of behavior gates. It can also be observed that LSTM-Medium with behavior gating gives similar performance to baseline LSTM-Large even though the latter has more than three times the number of parameters.