Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
Table 3: Tagging accuracy on , using the full PTB tagset to compare with yang2016part.
['System', 'Accuracy', 'In-vocab', 'Out-of-vocab']
AdaptaBERT outperforms task-tuned BERT by 3.9%, again due to improvements on OOV terms. Task-tuned BERT is on par with the best previous unsupervised domain adaptation result (FEMA), showing the power of contextualized word embeddings, even across disparate domains. Note also that FEMA’s representation was trained on the entire \ppceme, including the unlabeled test set, while the AdaptaBERT model uses the test set only for evaluation.