GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Table 3: Performance of MoE models with different number of experts and layers.
['Id', 'Model', 'Cores', 'Steps per sec.', 'Batch sz. (Tokens)', 'TPU core years', 'Training time (days)', 'BLEU avg.']
For each configuration shown in Fig. (E) fixed, brings consistent gains for both low and high resourced languages (upwards Δ shift along the y-axis), almost with a constant additive factor every time we scale the depth from 12L to 36L