Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Table 7: Model comparison on 1 Billion Word Language Modeling Benchmark. Models marked with * are from (Jozefowicz et al., 2016).
['Model', 'Test Perplexity', 'Test Perplexity', 'ops/timestep (millions)', '#Params excluding embed. & softmax', 'Total #Params', '[ITALIC] Drop-  [ITALIC] Prob', 'TFLOPS per GPU']
For our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.74-0.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA.