CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
Table 2: Statistical information of the two versions of the dictionaries. “Special Tokens” include “[PAD]”, “[UNK]”, “[CLS]”, “[SEP]”, “[MASK]”, “⟨S⟩”, “⟨T⟩” and 99 unused tokens.
['Token Type', 'Google', 'CLUE']
The original BERT model uses character-based tokenization for Chinese. But there are many redundant tokens in the original vocabulary. Therefore, we have compiled a refined vocabulary through automated scripts and manual review. We remove many unnecessary tokens, which will not be used in most cases of Chinese NLP tasks, such as traditional Chinese, Japanese, Korean and emojis. For English, we remove the most prefix tokens except for the single character and retain the most suffix tokens to guarantee the tokenization for English words. Similarly, for tokens standing for numbers, we only keep separate numeric symbols and the more commonly used words that represent the year. In addition, we also remove tokens composed of more than two special symbols.