KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Table 8: GLUE results on the dev set (%).
['Model', 'MNLI (m/mm)', 'QQP', 'QNLI', 'SST-2']
We notice that KEPLER keeps a consistent result with its baseline on large datasets like MNLI, but suffers an unstable performance on small datasets like RTE. In general, KEPLER achieves comparable results with RoBERTa∗ on GLUE, which suggests that the joint training in KEPLER does not harm the general language understanding ability.