Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
Table 2: Tagging accuracy on , using the coarse-grained tagset. The unsupervised systems never see labeled data in the target domain of Early Modern English. † in line 4, “in-vocab” and “out-of-vocab” refer to the training set vocabulary; for lines 1-3, this refers the PTB training set.
['System', 'Early Modern English Accuracy', 'Early Modern English In-vocab', 'Early Modern English Out-of-vocab', 'PTB Accuracy']
Task-tuning improves accuracy by 7.6% on the coarse-grained tagset (line 2), and domain-adaptive fine-tuning yields a further 4.5% in accuracy (line 3). AdaptaBERT’s performance gains are almost entirely due to the improvement on out-of-vocabulary terms, as discussed below. This improvement is largely attributable to in-vocabulary terms: although fine-tuned BERT does better than AdaptaBERT on both IV and OOV terms, the IV terms are far more frequent. The most frequent errors for AdaptaBERT are on tags for \exampleto (5337), \exampleall (2054), and \examplethat (1792). The OOV term with the largest number of errors is \exampleal (306), which is a shortening of \exampleall.