Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
Table 1: Comparison of validation and test set perplexity for r-RNTNs with f mapping (K=100 for PTB, K=376 for text8) versus s-RNNs and m-RNN. r-RNTNs with the same H as corresponding s-RNNs significantly increase model capacity and performance with no computational cost. The RNTN was not run on text8 due to the number of parameters required.
['Method', '[ITALIC] H', 'PTB # Params', 'PTB Test PPL', 'text8 # Params', 'text8 Test PPL', 'Method', '[ITALIC] H', 'PTB # Params', 'PTB Test PPL']
It is remarkable that even with K as small as 100, the r-RNTN approaches the performance of the RNTN with a small fraction of the parameters. This reinforces our hypothesis that complex transformation modeling afforded by distinct matrices is needed for frequent words, but not so much for infrequent words which can be well represented by a shared matrix and a distinct vector embedding. It appears that heuristically allocating increased model capacity as done by the f based r-RNTN is a better way to increase performance than simply increasing hidden layer size, which also incurs a computational penalty. Although m-RNNs have been successfully employed in character-level language models with small vocabularies, they are seldom used in word-level models.