Pretrained Transformers Improve Out-of-Distribution Robustness
Table 2: We train and test models on different STS-B distributions (Images, MSR videos, Headlines, and MSR paraphrase). The severe drop in the Pearson correlation coefficient shows the consequence of a distribution shift. Models such as Average GloVe lose nearly all performance when out-of-distribution. RoBERTa does especially well in comparison to other models.
['Train', 'Test', 'BoW', 'Average word2vec', 'LSTM word2vec', 'ConvNet word2vec', 'Average GloVe', 'LSTM GloVe', 'ConvNet GloVe', 'BERT Base', 'BERT Large', 'RoBERTa']
For STS-B, ReCoRD, and SST-2/IMDb, there is a noticeable drop in accuracy when testing on OOD examples.