PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation
Table 3: F1 scores of PKUSEG with and without pre-training. Here are results without using dictionaries. Web data comes from the Weibo dataset.
['[EMPTY]', 'w/o Pre-train', 'w. Pre-train']
We combine existing large-scale datasets together, including PKU (news), Weibo (web), and CTB8 (hybrid), and use them as pre-training data to obtain a coarse-grained model. Then the coarse-grained model is used to fine-tune domain-specific models. The pre-training models perform much better in terms of average score, especially on datasets with lower resource (e.g., tourism).