To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
Table 6: Accuracy of feature extraction () and fine-tuning () with BERT-base trained on training data of different MNLI domains and evaluated on corresponding dev sets. te: telephone. fi: fiction. tr: travel. go: government. sl: slate.
['[EMPTY]', 'te', 'go', 'tr', 'fi', 'sl']
S5SS0SSS0Px4 Impact of target domain Pretrained language model representations are intended to be universal. However, the target domain might still impact the adaptation performance. We find no significant correlation. At least for this task, the distance of the source and target domains does not seem to have a major impact on the adaptation performance.