Revisiting Pre-Trained Models for Chinese Natural Language Processing
Table 9: Ablations on MacBERT on different down-stream tasks. We also include average scores of all metrics.
['[EMPTY]', '[BOLD] CMRC 2018  [BOLD] EM', '[BOLD] CMRC 2018  [BOLD] F1', '[BOLD] DRCD  [BOLD] EM', '[BOLD] DRCD  [BOLD] F1', '[BOLD] CJRC  [BOLD] EM', '[BOLD] CJRC  [BOLD] F1', '[BOLD] XNLI  [BOLD] ACC', '[BOLD] LC  [BOLD] ACC', '[BOLD] BQ  [BOLD] ACC', '[BOLD] CSC  [BOLD] ACC', '[BOLD] THUC  [BOLD] ACC', '[BOLD] AVG']
We carried out ablations to examine the contributions of each component in MacBERT, which was thoroughly evaluated in all fine-tuning tasks. From a general view, removing any component in MacBERT will result in a decline in the average performance, suggesting that all modifications contribute to the overall improvements. Specifically, the most effective modifications are the N-gram masking and similar word replacement, which are the modifications on the masked language model task. When we compare N-gram masking and similar word replacement, we could see clear pros and cons, where N-gram masking seems to be more effective in text classification tasks, and the performance of reading comprehension tasks seems to benefit more from the similar word replacement task. By combining these two tasks will compensate each other, and have a better performance on both genres.