Long Short-Term Sample Distillation
Table 2: GLUE results (%) obtained by BERT and CNN, the metric of RTE MPRC, and SST-2 is accuracy, and the metrics of CoLA is Matthew’s Corr. Bold values indicate the best performance.
['[BOLD] Network', '[BOLD] Method', '[BOLD] RTE', '[BOLD] MRPC', '[BOLD] SST-2', '[BOLD] CoLA']
Specifically, when applied to BERT, LSTSD achieves accuracies of 74.73%, 89.22%, 93.35% on RTE, MRPC, and SST-2, respectively, outperforming all other baselines and vanilla BERT. It achieves a Matthew’s Correlation of 61.59% on CoLA, which is comparable with Mean Teacher. Similarly, when applied to CNN, LSTSD outperforms all baselines and vanilla CNNs on RTE, SST-2, CoLA, and is comparable to Snapshot Distillation on MRPC. It is worth mentioning that BERT and CNN are substantially different architectures, since the core of BERT is an attention mechanism, while the core of CNN are convolutions. Despite the great difference between BERT and CNN, LSTSD achieves consistent gains with both of them, which further establishes the generalization ability of LSTSD.