On Learning Universal Representations Across Languages
Table 3: GLUE dev results. Results with † and ‡ are from (Liu et al., 2019) and our in-house implementation respectively. We compare the performance of Hictl to BERTLarge, XLNetLarge (Yang et al., 2019), RoBERTa (Liu et al., 2019) and XLM-RLarge on the English GLUE benchmark.
['Model', '#Params', '[BOLD] QQP', '[BOLD] QNLI', '[BOLD] SST-2', '[BOLD] MRPC', '[BOLD] MNLI', '[BOLD] STS-B', '[BOLD] Avg']
We can observe that Hictl achieves significantly improvements, by 1.0% accuracy on average, over XLM-RBase on all tasks. Note that Hictl also outperforms XLM-RLarge and reaches performance on par with XLNetLarge with less parameters. These results demonstrate the effectiveness of learning universal representations at both sentence and word level for many languages, which maintains strong capabilities on per-language downstream tasks as well.