The Lottery Ticket Hypothesis forPre-trained BERT Networks
Table 2: Performance of subnetworks at the highest sparsity for which IMP finds winning tickets on each task. To account for fluctuations, we consider a subnetwork to be a winning ticket if its performance is within one standard deviation of the unpruned BERT model. Entries with errors are the average across five runs, and errors are the standard deviations. IMP = iterative magnitude pruning; RP = randomly pruning; θ0 = the pre-trained weights; θ′0 = random weights; θ′′0 = randomly shuffled pre-trained weights.
['Dataset', 'MNLI', 'QQP', 'STS-B', 'WNLI', 'QNLI', 'MRPC', 'RTE', 'SST-2', 'CoLA', 'SQuAD', 'MLM']
To account for fluctuations in performance, we consider a subnetwork to be a winning ticket if the performance of full BERT is within one standard deviation of the performance of the subnetwork. The highest sparsities at which we find these winning tickets range from 40% (SQuAD) and 50% (MRPC and CoLA) to 90% (QQP and WNLI). There is no discernible relationship between the sparsities for each task and properties of the task itself (e.g., training set size). Are IMP winning tickets sparser than randomly pruned or initialized subnetworks? Prior work describes winning tickets as a “combination of weights and connections capable of learning” (frankle2018the). That is, both the specific pruned weights and the specific initialization are necessary for a winning ticket to achieve this performance. To assess these claims in the context of BERT, we train a subnetwork f(x;mRP⊙θ0,⋅) with a random pruning mask (which evaluates the importance of the pruning mask mIMP) and a subnetwork f(x;mIMP⊙θ′0,⋅) with a random initialization (which evaluates the importance of the pre-trained initialization θ0). This confirms the importance of the specific pruned weights and initialization in this setting. The accuracy of random pruning is close to that of IMP only at the lowest sparsities (10% to 20%) if at all, confirming that the structure of the pruning mask is crucial for the performance of the IMP subnetworks at high sparsities.