ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Table 6: The results of 9 common Chinese NLP tasks. ERNIE 1.0 indicates model released by [sun2019ernie, ERNIE] . The reported results are the average of five experimental results, and the state-of-the-art results are in bold.
['Task', 'Metrics', 'BERT [ITALIC] BASE Dev', 'BERT [ITALIC] BASE Test', 'ERNIE 1.0 [ITALIC] BASE Dev', 'ERNIE 1.0 [ITALIC] BASE Test', 'ERNIE 2.0 [ITALIC] BASE Dev', 'ERNIE 2.0 [ITALIC] BASE Test', 'ERNIE 2.0 [ITALIC] LARGE Dev', 'ERNIE 2.0 [ITALIC] LARGE Test']
It can be seen that ERNIE 1.0BASE outperforms BERTBASE on XNLI, MSRA-NER, ChnSentiCorp, LCQMC and NLPCC-DBQA tasks, yet the performance is less ideal on the rest, which is caused by the difference in pre-training between the two methods. Specifically, the pre-training data of ERNIE 1.0BASE does not contain instances whose length exceeds 128, but BERTBASE is pre-trained with the instances whose length are 512. From the results, it can be also seen that the proposed ERNIE 2.0 makes further progress, which significantly outperforms BERTBASE on all of the nine tasks. Furthermore, we train a large version of ERNIE 2.0. ERNIE 2.0LARGE achieves the best performance and creates new state-of-the-art results on these Chinese NLP tasks.