Why Do Masked Neural Language Models Still Need Common Sense Knowledge?
Table 1: Results of micro and macro average hits@K for the ConceptNet relations. The macro avg. equally average the results of all relations, while the micro avg. weighted average the results of the relations according to the their portion.
['Model', 'Model', 'Hits@K 1', 'Hits@K 5', 'Hits@K 10', 'Hits@K 100']
Here, we report micro- and macro-average over the relations. We report individual results on each relation in Appendix B. Indeed, large fluctuation can be found in the quantitative results for each relation. Some relations (‘DefinedAs’, ‘IsA’, …) show below 20% in hits@100 while some (‘NotCapableOf’, ‘MadeOf’, ‘ReceivesAction’) show above 70%. The macro-average displays that BERTlarge outperforms BERTbase while the micro-average shows that BERTbase has higher performance than BERTlarge except for a hits@5. This mainly comes from the inconsistency on data distributions between the training dataset of MNLMs and ConceptNet, where some relations occupying a large portion of ConceptNet are not trained more elaborately in the larger model whilst most relations are trained better in the larger model generally.