[
Table 3: Performance on the test datasets of GLUE. The metrics for MRPC and QQP are F1/accuracy. The metrics for STS-B are Pearson correlation and Spearman correlation.
['[EMPTY]', 'BERT', 'CERT', 'XLNet', 'ERNIE2', 'T5', 'RoBERTa', 'ALBERT']
Among the 11 tasks, CERT outperforms BERT on 7 tasks, including RTE, QNLI, STS-B, MRPC, MNLI-m, MNLI-mm, QQP. CERT achieves the same performance as BERT on WNLI and AX. CERT performs worse than BERT on CoLA and SST-2. CERT achieves an average score of 80.7, which is better than BERT. Overall, CERT performs better than BERT, which further demonstrates that contrastive self-supervised learning is an effective approach for learning better representations of language. While CERT achieves much better performance than BERT on the validation set of CoLA, it performs worse than BERT on the CoLA test set. This is probably because the test set and validation set in CoLA have a large domain difference. For the next step, we plan to replace the BERT-based sentence encoder in CERT with XLNet, ERNIE2, T5, RoBERTa, and ALBERT, to see whether CSSL pretraining can improve the performance of these encoders.