Efficient softmax approximation for GPUs
Table 1: Text8: perplexity and training time after 5 epochs. Our approach is significantly better than other published approximate strategies. We also show that improving the baseline D-softmax [*] as discussed in text improve the results, but is slower than our proposal. Note, approximate strategies are comparatively less interesting for small vocabularies such as in this case.
['[EMPTY]', 'ppl', 'training time']
S5SS0SSS0Px4 Comparison with the state of the art. On this small vocabulary, approximate methods are comparatively less interesting. Our approach is the only one to approach the result of the full soft-max (below by 3 points of perplexity), while being the fastest. Our improved variant D-softmax [*] of the work by Chen et al.