Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
Table 5: Comparison on the DRCD dev set. The nT denotes the number of additional transformer layers.
['Model', 'F1', 'Param. Number']
+2T means additional 2 layers, M denotes million. As the experimental results showed, when the number of additional layers exceeds 1, the performance starts to decline, which demonstrates that using an extensive model on top of the PLM representations may not bring additional benefits. We can conclude that MWA doesn’t introduce too many parameters, and MWA achieves better performance than +1T under the similar parameter numbers. Besides, we also make comparison with the current best Chinese PLM: Robust-BERT-wwm-ext-large BERTwwmreport, a 24-layers Chinese PLM with 13.5 times more pre-training data and 3.1 times more parameters than BERT-wwm, experimental results show that our model can achieve comparable performance, which again confirms the effectiveness of incorporating word segmentation information into character-level PLMs.