Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
Table 4: F1 results of ablation experiments on the DRCD dev set.
['Model', 'BERT', 'BERT-wwm', 'ERNIE']
To demonstrate the effectiveness of our multi-source fusion method, we carry out experiments on the DRCD dev set with different segmentation inputs. Besides, we also design two strong baselines by introducing a Transformer layer (1T) and a random tokenizer model (\text{WA}_{random}) to exclude the benefits from additional parameters. Compared with 1T and \text{WA}_{random}, our proposed word-aligned attention gives quite stable improvements no matter what CWS tool we use, which again confirms the effectiveness and rationality of incorporating word segmentation information into character-level PLMs. Another observation is that employing multiple segmenters and fusing them together could introduce richer segmentation information and further improve the performance.