Reducing Transformer Depth on Demand with Structured Dropout
Figure 7: Effect of LayerDrop on Training Time
['Model', 'Valid PPL']
A1SS3SSS0Px1 Impact of LayerDrop on training time. The words per second were computed on 8 V100 GPUs with 32GB of memory, without floating point 16, for a 16 layer model trained on Wikitext-103. Assuming fixed layer size, LayerDrop removes layers at training time randomly, which increases the training speed almost 2x if dropping half the number of layers.