1 Introduction
Table 4: Low-sparsity supermask performance. We report the sparsity levels achieved when the supermasks were initialized at 0% sparsity. For several tasks, fine-tuning is achieved with less than 3% of pre-trained weights pruned. For the supermask evaluation results, we include the mean and standard deviation of 10 Bernoulli samplings of a single run.
['GLUE Task', 'MNLI', 'QQP', 'QNLI', 'SST-2', 'CoLA', 'STS-B', 'MRPC', 'RTE']
Surprisingly, these low-sparsity supermasks successfully learn to perform all the tasks without noticeable degradation from baseline. Fine-tuning can indeed be very fine, suggesting relatively frequent occurrences of good solutions within a sparse L0-neighborhood of the pre-trained parameters. For MNLI, the GLUE task with the most fine-tuning steps, the sparsity level reaches 12.9%. We ask how prediction accuracy grows with sparsity during fine-tuning. This makes model performance increase roughly linearly with sparsity.