Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Table 8: Speedup obtained for the 1.2 billion parameters model using model parallelism while keeping the batch size constant.
['# of GPUs', '1', '2', '4', '8']
Our model parallelism is primarily designed to enable training models larger than what can fit in the memory of a single GPU, but it can also accelerate the training of smaller models without increasing the batch size. To measure this acceleration we train a model with a fixed 1.2 billion parameters. We use a fixed batch size of 8 samples per iteration and increase the number of GPUs using model parallelism. Using two GPUs makes training 64% faster. Above that we see diminishing returns as the per-GPU computation decreases and the memory bandwidth and communication overheads begin to dominate.