The Lottery Ticket Hypothesis forPre-trained BERT Networks
Table 3: Performance of subnetworks found using IMP with rewinding to the steps in the left column and standard pruning (where subnetworks are trained using the final weights from the end of training).
['Dataset', 'MNLI', 'QQP', 'STS-B', 'WNLI', 'QNLI', 'MRPC', 'RTE', 'SST-2', 'CoLA', 'SQuAD', 'MLM']
In settings where winning tickets have not been found, IMP still finds matching subnetworks at non-trivial sparsities. However, these subnetworks must be initialized to the state of the network θi after i steps of training (frankle2019lottery; morcos2019one; yu2019playing) rather than to initialization θ0. This procedure is known as rewinding, since the subnetwork found by IMP is rewound to the weights at iteration i rather than reset to initialization θ0. Although we have already found winning tickets, we still evaluate the possibility that rewinding improves performance. For example, on SQuAD v1.1, i∈{1000,2000,4000,10000}. Claim 4: Do IMP subnetworks match the performance of standard pruning? In the limit of rewinding, we can initialize the IMP subnetwork to weights θt at the end of training, producing a subnetwork f(x;m⊙θt,⋅). Doing so is the basis of a standard way of pruning neural networks: train the network to completion, prune, and train further to recover performance without rewinding (han2015deep). Renda et al. (Renda2020Comparing) show that, in other settings, IMP subnetworks rewound early in training reach the same accuracies at the same sparsities as subnetworks found by this standard pruning procedure. For some tasks (QQP, QNLI, MRPC, MLM), standard pruning improves upon winning ticket performance by up to two percentage points. For others (STS-B, WNLI, RTE, SST-2), performance drops by up to three percentage points. The largest drops again occur for tasks with small training sets where standard pruning may overfit.