Neural Chinese Word Segmentation with Lexicon and Unlabeled Data via Posterior Regularization
Table 3. Experimental results of domain adaptation.
['[EMPTY]', 'Zhuxian', 'Zhuxian', 'Zhuxian', 'Weibo', 'Weibo', 'Weibo']
Two datasets are used in our experiments. This dataset contains 20,135 sentences for training and 8,592 sentences for test. Since there is no off-the-shelf lexicon for Weibo word segmentation, we built one using the words extracted from the training data. In addition, we regarded the training sentences as unlabeled data. We used Zhuxian and Weibo datasets as two target domains, and used the UPUC dataset as source domain. The settings of our approach and baseline methods are consistent with previous experiments. This is because there is huge difference in word distribution between source and target domains. Many words in target domain may not or rarely appear in the labeled data of source domain, making it difficult for these CWS methods to segment the sentences in target domain. Our approach can outperform these methods because beyond the lexicon information our approach can also incorporate massive unlabeled data into neural model training, which can provide useful information for CWS. This result implies that incorporating the lexicon and unlabeled data as indirect supervision via posterior regularization is more suitable for domain adaptation of CWS than utilizing them to build partially labeled data through word matching, which may introduce heavy noise.