BERT Loses Patience:Fast and Robust Inference with Early Exit
Table 2: Experimental results (median of 5 runs) of BERT based models on the development set of GLUE. We mark “-” on STS-B for BranchyNet and Shallow-Deep since they do not support regression.
['[BOLD] Method', '[BOLD] #Param', '[BOLD] Speed', '[BOLD] MNLI', '[BOLD] SST-2', '[BOLD] STS-B']
We can see that our BERT-based model significantly outperforms other compared methods with either knowledge distillation or prediction probability based input-adaptive inference methods. Notably, the performance is slightly lower than the original BERT model while PABEE improves the accuracy on ALBERT. We suspect that this is because the intermediate layers of BERT have never been connected to an output layer during pretraining, which leads to a mismatch between pretraining and fine-tuning when adding the internal classifiers. However, PABEE still has a higher accuracy than various knowledge distillation-based approaches as well as prediction probability distribution based models, showing its potential as a generic method for deep neural networks of different kinds. Although more classifiers need to be trained, training PABEE is no slower (even slightly faster) than conventional fine-tuning, which may be attributed to the additional loss functions of added internal classifiers. This makes our approach appealing compared with other approaches for accelerating inference such as pruning or distillation because they require separately training another model for each speed-up ratio in addition to training the full model. Also, PABEE only introduces fewer than 40K parameters (0.33% of the original 12M parameters).