Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
(b)
['[BOLD] Type', '[BOLD] Batch', '[BOLD] LR', '[BOLD] Time', '[BOLD] BPC', '[BOLD] SST', '[BOLD] IMDB']
We find that moving to mixed precision not only achieves similar training results, but it also provides a 3x speedup in training. By taking advantage of the reduced memory footprint of FP16 storage, we increase the batch size two-fold to 256 per GPU, better saturating the GPU, and achieve an additional speedup of 40% on top of our original speedup. This provides approximately a 4.2x speedup when switching from single precision arithmetic to mixed precision. We were able to converge our model in mixed precision, to a similar value as the FP32 baseline. This speeds up training, and substantially reduces our memory footprint, without a measurable change in accuracy, as shown in Fig. We further speed up training by saturating up to 128 GPUs with distributed data parallelism, which we can do with a near-linear scaling factor Fig.