Slim Embedding Layers for Recurrent Neural Language Models
Table 2: Perplexity results for single models on BillionW. Bold number denotes results on a single GPU.
['Model', 'Perplexity', '#P[Billions]']
In the one billion word experiments, the total memory during training used on the GPU is about 7GB, and is smaller if a larger compression rate is used. We use a fixed smoothed unigram distribution (unigram distribution raised to 0.75) as the noise distribution. For the two layer model, the compression rate for the input layer is 1/32 and the output layer is 1/8, and the total number of parameters is 322 million. For the three layer model, the compression rates for the input and output layer are 1/32 and 1/16, and the total number of parameters is 254 million. Both experiments using NCE take about seven days of training on a GTX 1080 GPU. \citeauthorjozefowicz2016exploring \shortcitejozefowicz2016exploring suggests importance sampling (IS) could perform better than the NCE model, so we ran the experiment using IS and we used 4000 noise samples for each mini-batch. The PPL decreased to 38.3 after training for 8 days. As far as we know, the 3 layer model is the most compact recurrent neural language model that has a perplexity below 40 on this dataset.