Structured Pruning of a BERT-based Question Answering Model
Table 2: Decoding times, accuracies, and space savings achieved at sample operating points on large-qa
['model', 'time (sec)', '[ITALIC] f1', '% attn removed', '% ff removed', 'size (MiB)', '[ITALIC] f1 no retrain']
4.2.3 Validating these results On the basis of the development experiments, we select an operating point, namely the largest values of λattn and λff with <5 F1-point loss. We train the feed-forward and attention gates of large-qa with these penalties, as well as multiples 2×, 3×, and 4×. Accuracies are medians of 5 seeds, and timings are medians of 5 decoding runs with the median seed, on a single Nvidia K80 with batch size 1. Models in which both attention and feed-forward components are pruned were built from the independently trained gate configurations of attention and feed forward. For corresponding penalty weights, the large-qa was pruned somewhat less than base-qa, and the F1 loss due to pruning was smaller.