How Can We Know What Language Models Know?
Table 8: Micro-averaged accuracy (%) of various LMs
['[BOLD] Model', '[BOLD] Man', '[BOLD] Mine', '[BOLD] Mine  [BOLD] +Man', '[BOLD] Mine  [BOLD] +Para', '[BOLD] Man  [BOLD] +Para']
, we compare BERT with ERNIE and KnowBert, which are enhanced with external knowledge by explicitly incorporating entity embeddings. ERNIE outperforms BERT by 1 point even with the manually defined prompts, but our prompt generation methods further emphasize the difference between the two methods, with the highest accuracy numbers differing by 4.2 points using the Mine+Man method. This indicates that if LMs are queried effectively, the differences between highly performant models may become more clear. KnowBert underperforms BERT on LAMA, which is opposite to the observation made in Peters et al. This is probably because that multi-token subjects/objects are used to evaluate KnowBert in Peters et al.