SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Table 5: Hyperparameters selected by our hyperparameter search when training SqueezeBERT with bells-and-whistles.
['Hyperparameter', 'MNLI-m', 'MNLI-mm', 'QQP', 'QNLI', 'STS-2', 'CoLA', 'STS-B', 'MRPC', 'RTE']
We observe two interesting phenomena on this table. The first is regarding the use of distillation. When α=1.0, the teacher logits are ignored, and thus distillation is disabled. The second interesting phenomenon in this table is that maximum accuracy was not necessarily achieved on final epoch. For example, on QNLI, SqueezeBERT converged to its maximum development-set accuracy after just two epochs.