CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
Table 2: XNLI dev accuracy for English, Russian, Chinese and Urdu (Δ for average) for BERT-BASE models trained either on Wikipedia or CommonCrawl. The additional data provided by our pipeline alleviates the lack of resources in most languages and enables representation learning for low-resource languages such as Urdu.
['[EMPTY]', 'en', 'ru', 'zh', 'ur', 'Δ']
We also train BERT models on the English (en), Russian (ru), Chinese (zh) and Urdu (ur) languages, using either the Wikipedia corpora or our new CommonCrawl datasets. For these languages, we use respectively 16G, 5G, 1.1G and 106M of raw Wikipedia data (full datasets), and we cap the head CommonCrawl data to 21G, 21G, 17G, 2.2G for English, Russian, Chinese and Urdu. That is, we consider roughly the same amount of data for English, but increase the amount of data for Russian, Chinese and Urdu. We train a BERT-BASE architecture [devlin2018bert] on each of these corpora, without next sentence prediction (NSP) as in [lample2019cross]. For better comparison, we early-stop all our models after two days of training on 16 Volta32 GPUs, and use the exact same number of steps for each model. We evaluate each model on the XNLI [conneau2018xnli] corpus by using the training data in each language. With the same amount of data for English, the BERT-BASE model trained on our corpus outperforms the one trained on the Wikipedia. For low-resource languages like Urdu (ur), the Wikipedia dataset being too small, the model pretrained on Wikipedia obtains similar performance than a randomly initialized model. Using our corpus instead, we obtain a 7 points improvement in accuracy, which demonstrates how our filtered corpus can enable language model pretraining for low-resource languages.