ZEN: Pre-training Chinese Text EncoderEnhanced by N-gram Representations
Table 3: The performance of BERT and ZEN on seven NLP tasks when they are trained on a small corpus.
['[EMPTY]', 'CWS Test', 'POS Dev', 'POS Test', 'NER Test', 'DC Dev', 'DC Test', 'SA Dev', 'SA Test', 'SPM Dev', 'SPM Test', 'NLI Dev', 'NLI Test']
Pre-trained models usually require a large corpus to perform its training. However, in many applications in specialized domains, a large corpus may not be available. For such applications with limited training data, ZEN, with n-gram enhancement, is expected to encode text much more effectively. Therefore, to further illustrate the advantage of ZEN, we conduct an experiment that uses a small corpus to pre-train BERT and ZEN. In detail, we prepare a corpus with 1/10 size of the entire Chinese Wikipedia by randomly selecting sentences from it. Then all encoders are pre-trained on it with random initialization and tested on the same NLP tasks in the previous experiment. In general, same trend is shown in this experiment when compared with that in the previous one, where ZEN constantly outperform BERT in all task. This observation confirms that representing n-grams provides stable enhancement when our model is trained on corpora with different sizes. In detail, these results also reveals that n-gram information helps more on some tasks, e.g., CWS, NER, NLI, over the others. The reason is not surprising since that boundary information carried by n-grams can play a pivotal role in these tasks. Overall, this experiment simulates the situation of pretraining a text encoder with limited data, which could be a decisive barrier in pre-training a text encoder in the cold-start scenario, and thus demonstrates that ZEN has its potential to perform well in this situation.