Using the Output Embedding to Improve Language Models
Table 5: Word level perplexity (lower is better) on PTB and size (number of parameters) of models that use either dropout (baseline model) or Bayesian dropout (BD). WD – weight decay.
['Model', 'Size', 'Train', 'Val.', 'Test']
In the first experiment, which was conducted on the PTB dataset, we compare the perplexity obtained by the large NNLM model and our version in which the input and output embeddings are tied. As can be seen in Tab. This indicates less overfitting, as expected due to the reduction in the number of parameters. They obtained improved performance. When the embeddings of this model are tied, a similar amount of improvement is gained. We tried this with and without weight decay and got similar results in both cases, with slight improvement in the latter model. The contribution of WT is also significant in this model.