BERT Loses Patience:Fast and Robust Inference with Early Exit
Table 5: Results on the adversarial robustness. “Query Number” denotes the number of queries the attack system made to the target model and a higher number indicates more difficulty.
['[BOLD] Metric (↑ better)', '[BOLD] ALBERT SNLI', '[BOLD] ALBERT MNLI-m/-mm', '[BOLD] ALBERT Yelp', '[BOLD] + Shallow-Deep\xa0kaya2018shallow SNLI', '[BOLD] + Shallow-Deep\xa0kaya2018shallow MNLI-m/-mm', '[BOLD] + Shallow-Deep\xa0kaya2018shallow Yelp', '[BOLD] + PABEE  [ITALIC] (ours) SNLI', '[BOLD] + PABEE  [ITALIC] (ours) MNLI-m/-mm', '[BOLD] + PABEE  [ITALIC] (ours) Yelp']
Our approach successfully defends more than 3× attacks compared to the original ALBERT on NLI tasks, and 2× on the Yelp sentiment analysis task. Also, PABEE increases the number of queries needed to attack by a large margin, providing more protection to the model. Compared to Shallow-Deep kaya2018shallow, our model demonstrates significant robustness improvements. To analyze, although the early exit mechanism of Shallow-Deep can prevent the aforementioned overthinking problem, it still relies on a single classifier to make the final prediction, which makes it vulnerable to adversarial attacks. In comparison, since Patience-based Early Exit exploits multiple layers and classifiers, the attacker has to fool multiple classifiers (which may exploit different features) at the same time, making it much more difficult to attack the model. This effect is similar to the merits of Ensemble Learning against adversarial attack, discussed in previous studies strauss2017ensemble; tramer2018ensemble; pang2019improving.