CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
Table 4: The effects of more training data of C5 and steps. With three times steps(3* 125k), BERT-base model trained on C5, gain 0.47 to 0.82 point compare to same model trained with 125k using different vocabularies. With tree times training data(3* 1GB), BERT-base model gain 0.74 to 1.07 point compare to same model trained with 1GB data and 125K, 0.25 to 0.27 point compare to same steps on 1GB. And the model trained with CLUE vocab always better than with Google vocab.
['Index', 'Model', 'Vocab', 'Data', 'Steps', 'AFQMC', 'TNEWS', 'IFLYTEK', 'CMNLI', 'AVG']
Meanwhile, the performance of BERT-base which use clue_vocab is 0.1 point higher than the performance of BERT-base which use google vocabulary through Index 5 and 6. It can be concluded that the increase of the corpus can improve the performance of the model, and our clue_vocab is better than google vocabulary while BERT-base model is trained enough steps.