GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Table 2: The number of tokens have been seen by a model during training to reach three different cross-entropy loss. A general trend is that deeper models are more sample efficient and converge faster than the comparable shallow ones.
['Id', 'Model', 'Cores', 'Billion tokens to cross-entropy of 0.7', 'Billion tokens to cross-entropy of 0.6', 'Billion tokens to cross-entropy of 0.5']
For this purpose, we compare number of tokens being processed by each model to reach a preset training loss. For example MoE(128E, 12L) takes 3 times the number of tokens to reach 0.7 training cross-entropy compared to MoE(128E, 36L), (6) vs (5). We observe a similar trend for models with 512 and 2048 experts, (4) vs (3) and (2) vs (1). Comparing the models with same depth, (5), (3) and (1), we notice a significant drop in the number of tokens required to reach training loss of 0.7, as we transition from 128 to 512 number of experts. After this phase shift, models with ample capacity tend to exhibit similar sample efficiency characteristics, as in models (3) and (1).