1 Introduction
Table 6: Comparison between weights pruned with low-sparsity supermasks (initialized at 0% sparsity) and weights pruned with magnitude-based pruning at the same final sparsity. We report the maximum and mean magnitude of the pruned weights. The last row shows percentages of the overlap between the supermask and the magnitude-based pruning mask, i.e. the percentages of weights zeroed by the supermask that are also the smallest weights.
['GLUE task', 'MNLI', 'QQP', 'QNLI', 'SST-2', 'CoLA', 'STS-B', 'MRPC', 'RTE']
How does the learning of a supermask actually work? Does a supermask simply learn to prune away the weights with smallest magnitudes? Since pure magnitude-based pruning of pre-trained weights does not perform any task-specific learning, we reason that the weight entries being set to zero by the supermask must have significant values. Here, we inspect the magnitudes of the pre-trained weights zeroed by the supermasks These weights turn out to have remarkably higher magnitudes than the smallest entries, suggesting the learning of supermasks is not trivial magnitude-based pruning.