Pre-training Polish Transformer-based Language Models at Scale
Table 1: Results on the KLEJ benchmark.
['[BOLD] Model', '[BOLD] Average', 'NKJP', 'CDSC-E', 'CDSC-R', 'CBD', 'PE2-I', 'PE2-O', 'DYK', 'PSC', 'AR']
In this section, we demonstrate the results of evaluating our language models on downstream tasks. We repeated the fine-tuning of the models for each task five times. The scores reported are the median values of those five runs. The results of other approaches are taken from the KLEJ leaderboard. We split the table into two sections, comparing the BERT-base and BERT-large architectures separately. We can observe that there is a wider selection of base models, and most of them are multilingual, such as the original multilingual BERT (mBERT) [devlin-etal-2019-bert], SlavicBERT [arkhipov-etal-2019-tuning], XLM [NIPS2019_8928], and XLM-R [conneau2019unsupervised]. The only models pre-trained specifically for Polish language are HerBERT [rybak2020klej] and Polbert. Among the base models, our approach outperforms others by a significant margin. In the case of large models, only the XLM-RoBERTa (XLM-R) pre-trained model has been available until now. XLM-RoBERTa is a recently published multilingual transformer trained on 2.5TB of data in 100 languages. It has been shown to be highly competitive against monolingual models. A direct comparison with our Polish language model demonstrates a consistent advantage of our model - it has achieved better results in seven of the nine tasks included in the KLEJ benchmark.