Input-to-Output Gate to Improve RNN Language Models
Table 2: Comparison between baseline models and the proposed method (represented as “+ IOG”) on the Penn Treebank (PTB) dataset. † denotes results published in previous studies. The method with WT shared word embeddings (E in the Equation (5)) with the weight matrix of the final layer (W in the Equation (3)). AL denotes that the method used a previously proposed augmented loss function Inan et al. (2016).
['Model', 'Parameters', 'Validation', 'Test']
These tables, which contain both the scores reported in the previous studies and those obtained by our reproduced models, indicate that IOG reduced the perplexity. In other words, IOG boosted the performance of the baseline models. We emphasize that IOG is not restricted to a neural architecture of a language model because it improved the RHN and LSTM performances. \newciteDBLP:journals/corr/MerityXBS16 and \newciteDBLP: journals/corr/GraveJU16 proposed similar methods. Their methods, which are called “cache mechanism” (or ‘pointer’), keep multiple hidden states at past timesteps to select words from previous sequences. \newciteDBLP: journals/corr/InanKS16 and \newcitepress-wolf:2017: \newciteDBLP: \newcite45826 adopted RNNs and reinforcement learning to automatically construct a novel RNN architecture. We expect that IOG will improve these models since it can be combined with any RNN language models. Model ensemble techniques are widely used for further improving the performance of neural networks. In this experiment, we employed a simple ensemble technique: using the average of the output probability distributions from each model as output. We computed the probability distribution Pt+1 on the ensemble of the M models as follows: Pt+1=1MM∑m=1mPt+1, (9) where mPt+1 represents the probability distribution predicted by the m-th model. In the ensemble setting, we applied only one IOG to the multiple models. In addition, as additional experiments, we incorporated IOG with the latest method, which was proposed after the submission deadline of IJCNLP 2017. \newcitemerityRegOpt introduced various regularization and optimization techniques such as DropConnect Wan et al. They called their approach AWD LSTM, which is an abbreviation of averaged stochastic gradient descent weight-dropped LSTM. These tables show that IOG was not effective to AWD LSTM. Perhaps, the reason is that the perplexity of AWD LSTM is close to the best performance of the simple LSTM architecture. We also note that IOG did not have any harmful effect on the language models because it maintained the performances of AWD LSTM with ‘WT’ and the cache mechanism. Moreover, incorporating IOG is much easier than exploring the best regularization and optimization methods for each RNN language model. Therefore, to improve the performance, we recommend combining IOG before searching for the best practice.