[
Table 2: Performance on the validation dataset. The metric for MRPC is F1. The metric for QQP is accuracy. The metric for STS-B is Pearson correlation.
['[EMPTY]', 'CoLA', 'RTE', 'QNLI', 'STS-B', 'MRPC', 'WNLI', 'SST-2', 'MNLI (m/mm)', 'QQP']
Following (lan2019albert), we perform 5 random restarts of the finetuning and report the median and best performance among the 5 runs. Due to randomness of the 5 restarts, our median performance is not the same as that reported in (lan2019albert). But generally they are close. From this table, we make the following observations. First, in terms of median performance, our proposed CERT outperforms BERT (our run) on 5 tasks, including CoLA, RTE, QNLI, SST-2, and QQP. On the other 5 tasks, CERT is on par with BERT. This demonstrates the effectiveness of CERT in learning better language representations via contrastive self-supervised learning. Second, in terms of best performance, CERT outperforms BERT on 7 tasks, including CoLA, RTE, QNLI, STS-B, MNLI-m, MNLI-mm, and QQP. CERT is on par with BERT on MRPC and WNLI. CERT performs less well than BERT on SST-2. These results further demonstrate the effectiveness of CERT. Third, the improvement of CERT over BERT is more significant on tasks where the training data is small, such as CoLA and RTE. One possible reason is that small-sized training data is more prone to overfitting, which makes the necessity of CSSL pretraining more prominent. Fourth, on small-sized datasets such as CoLA and RTE, CERT achieves great improvement over BERT. This shows that CERT is promising in solving low-resource NLP tasks where the amount of training data is limited.