Are Sixteen Heads Really Better than One?
Table 4: Average inference speed of BERT on the MNLI-matched validation set in examples per second (± standard deviation). The speedup relative to the original model is indicated in parentheses.
['Batch size', '1', '4', '16', '64']
Moreover, we find that actually pruning the heads (and not just masking) results in an appreciable increase in inference speed. Experiments were conducted on two different machines, both equipped with GeForce GTX 1080Ti GPUs. Each experiment is repeated 3 times on each machine (for a total of 6 datapoints for each setting). We find that pruning half of the model’s heads speeds up inference by up to ≈17.5% for higher batch sizes (this difference vanishes for smaller batch sizes).