Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Table 8: Model comparison on 100 Billion Word Google News Dataset
['Model', 'Test Perplexity', 'Test Perplexity', 'ops/timestep (millions)', '#Params excluding embed. & softmax', 'Total #Params', 'TFLOPS per GPU']
: We evaluate our model using perplexity on a holdout dataset. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs.