Pretrained Transformers Improve Out-of-Distribution Robustness
Table 4: We train and test models on SST-2 and IMDB. Notice IID accuracy is not perfectly predictive of OOD accuracy, so increasing IID benchmark performance does not necessarily yield superior OOD generalization.
['Train', 'Test', 'BoW', 'Average word2vec', 'LSTM word2vec', 'ConvNet word2vec', 'Average GloVe', 'LSTM GloVe', 'ConvNet GloVe', 'BERT Base', 'BERT Large', 'RoBERTa']
For STS-B, ReCoRD, and SST-2/IMDb, there is a noticeable drop in accuracy when testing on OOD examples.