Pretrained Transformers Improve Out-of-Distribution Robustness
Table 3: For ReCoRD, the exact match performance is closely tethered to the test dataset, which suggests a difference in the difficulty of the two test sets. This gap can be bridged by larger Transformer models pretrained on more data.
['Train', 'Test', 'Document QA', 'DistilBERT', 'BERT Base', 'BERT Large', 'RoBERTa']
For STS-B, ReCoRD, and SST-2/IMDb, there is a noticeable drop in accuracy when testing on OOD examples.