ZEN: Pre-training Chinese Text EncoderEnhanced by N-gram Representations
Table 2: The overall performance of ZEN and the comparison against existing models on seven NLP tasks, where R denotes that pre-training starts from random initialization and P is that model parameters are initialized from Google’s released Chinese BERT base model. B and L refer to each backbone model uses BERT base or large model, respectively. Since ZEN uses BERT base model, encoders using BERT large model and their performance are listed as references in italic fonts. The bold numbers are the best results from all base models in each column.
['[EMPTY]', 'CWS Test', 'POS Dev', 'POS Test', 'NER Test', 'DC Dev', 'DC Test', 'SA Dev', 'SA Test', 'SPM Dev', 'SPM Test', 'NLI Dev', 'NLI Test']
The first experiment is to compare ZEN and BERT with respect to their performance on the aforementioned NLP tasks. In this experiment, ZEN and BERT use two settings, i.e., training from (R): randomly initialized parameters and (P): pre-trained model, which is the Google released Chinese BERT base model. Overall, in both R and P settings, ZEN outperforms BERT in all seven tasks, which clearly indicates the advantage of introducing n-grams into the encoding of character sequences. This observation is similar to that from dos2014deep; lample2016neural; bojanowski2017enriching; liu2019encoding. In detail, when compare R and P settings, the performance gap between ZEN (P) and BERT (P) is larger than that in their R setting, which illustrates that learning an encoder with reliable initialization is more important and integrating n-gram information contributes a better enhancement on well-learned encoders. For two types of tasks, it is noticed that token-level tasks, i.e., CWS, POS and NER, demonstrate a bigger improvement of ZEN over BERT than that of sentence-level tasks. where the potential boundary information presented by n-grams are essential to provide a better guidance to label each character. Particularly for CWS and NER, these boundary information are directly related to the outputs. Similarly, sequence-level tasks show a roughly same trend on the improvement of ZEN over BERT, which also shows the capability of combining both character and n-gram information in a text encoder. The reason behind this improvement is Such encoders include BERT-wwm (cui2019pre), ERNIE 1.0 (sun2019ernie), ERNIE 2.0 (B) (sun2019ernie2), ERNIE 2.0 (L) (sun2019ernie2), NEZHA (B) and (L) (wei2019nezha) where B and L denote the base and large model of BERT, respectively. Note that although there are other pre-trained encoders with exploiting entity knowledge or multi-model signals, they are not compared in this paper because external information are required in their work. In fact, even though without using such external information, ZEN still achieves the state-of-the-art performance on many of the tasks experimented.