Commonsense Knowledge Mining from Pretrained Models
Table 3: Test results examining the effect of sentence meaning and grammaticality on task performance. Scores are shown for a sample of 100 triples split by whether the generated sentence is grammatical and whether it conveys the correct meaning of the triple.
['[BOLD] Task 1', '[BOLD] N (/100)', '[BOLD] F1 Score']
In order to measure the impact of sentence generation on our model, we select a sample of 100 sentences and group the results by a) whether the sentence contained a grammatical error, and b) whether the sentence misrepresented the meaning of the triple. For example, the triple (golf, HasProperty, good) yields the sentence “golf is a good”, which is grammatically correct but conveys the wrong meaning. On both Wikipedia mining and CKBC, we find that misrepresenting meaning has an adverse impact on model performance. Future work could therefore focus on designing templates that more reliably encode a relation’s true meaning.