Mono vs Multilingual Transformer-based Models: a Comparison across Several Language Tasks
Table 9: Comparison of the 3 models across all tasks
['Model', 'BertPT >', 'BertPT =', 'BertPT ', 'AlbertPT >', 'AlbertPT =', 'AlbertPT ', 'Multilingual >', 'Multilingual =', 'Multilingual ']
The different tasks and settings under which BertPT and AlbertPT were compared to Multilingual BERT and published baselines yield 38 possible pairwise comparisons. We analyzed in how many times the performance of each model was better than, worse than or equivalent to the performance of the others. If the proportional difference in scores was below 5%, then the two models were considered equivalent. We can see that no single method is always better than the others. The baselines have fewer wins, which means that BERT-based models tend to yield better results. BertPT and AlbertPT achieved equivalent performances most of the time (28 out of 38). AlbertPT has an advantage (6 wins versus 4 losses), and since its training is less expensive, then it would be the model of choice.