BERT Loses Patience:Fast and Robust Inference with Early Exit
Table 1: Experimental results (median of 5 runs) of models with ALBERT backbone on the development set and the test set of GLUE. The numbers under each dataset indicate the number of training samples. The acceleration ratio is averaged across 8 tasks. We mark “-” on STS-B for BranchyNet and Shallow-Deep since they do not support regression.
['[BOLD] Method', '[BOLD] #Param', '[BOLD] Speed', '[BOLD] CoLA', '[BOLD] MNLI', '[BOLD] MRPC', '[BOLD] QNLI', '[BOLD] QQP', '[BOLD] RTE', '[BOLD] SST-2', '[BOLD] STS-B', '[BOLD] Macro']
This choice is made because: (1) ALBERT is a state-of-the-art PLM for natural language understanding. (2) ALBERT is already very efficient in terms of the number of parameters and memory use because of its layer sharing mechanism, but still suffers from the problem of high inference latency. We can see that our approach significantly outperforms all compared approaches on improving the inference efficiency of PLMs, demonstrating the effectiveness of the proposed PABEE mechanism. Surprisingly, our approach consistently improves the performance of the original ALBERT model by a relatively large margin while speeding-up inference by 1.57×. This is, to the best of our knowledge, the first inference strategy that can improve both the speed and performance of a fine-tuned PLM. In our experiments, we use a state-of-the-art adversarial attack method, TextFooler jin2019bert, which demonstrates effectiveness on attacking BERT. We conduct black-box attacks on three datasets: SNLI snli, MNLI mnli and Yelp yelp. We attack the original ALBERT-base model, ALBERT-base with Shallow-Deep kaya2018shallow and with Patience-based Early Exit.