Efficient softmax approximation for GPUs
Table 2: One Billion Word benchmark. Perplexity on the test set for single models. Our result is obtained after 5 epochs.
['Model', 'Test perplexity']
In comparison, only Jozefowicz et al. 8× bigger than ours and trained over 32 GPUs during 3 weeks. We also note that for models of similar size, we achieve similar perplexity than the method introduced by Jozefowicz et al. As far as we know, ours the first method to achieve a perplexity lower than 50 on a single GPU.