Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Table 6: Experiments with different combinations of losses.
['[ITALIC] wimportance', '[ITALIC] wload', 'Test Perplexity', '[ITALIC] CV( [ITALIC] Importance( [ITALIC] X))', '[ITALIC] CV( [ITALIC] Load( [ITALIC] X))', '[ITALIC] max( [ITALIC] Load( [ITALIC] X)) [ITALIC] mean( [ITALIC] Load( [ITALIC] X))']
All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.