Structured Pruning of a BERT-based Question Answering Model
Table 4: pruning RoBERTa on NQ, continued cross-entropy training on NQ
['model', '% attn', '% ff', 'LA', 'SA']
We pruned this RoBERTa-large NQ model, using the same techniques as described above, selecting the gate values by L0 regularization for one epoch on approximately 20% of the NQ training data, and continued training for epoch on the full NQ training set. For the same values of λattn and λff, the amount pruned is similar to BERT. The tradeoff loss in accuracy also shows a similar falloff.