ℂ𝕆𝕄𝔼𝕋: Commonsense Transformers for Automatic Knowledge Graph Construction
Table 3: Human evaluation testing effect of different decoding schemes on candidate tuple quality. The number of ratings made per relation for each decoding method is provided in the first column.
['ℂ𝕆𝕄𝔼𝕋  [BOLD] Decoding method', '[BOLD] oEffect', '[BOLD] oReact', '[BOLD] oWant', '[BOLD] xAttr', '[BOLD] xEffect', '[BOLD] xIntent', '[BOLD] xNeed', '[BOLD] xReact', '[BOLD] xWant', '[BOLD] Avg']
, we show the effect of different generation policies on knowledge quality. The most interesting result is that using greedy decoding to produce knowledge tuples only results in a 10% relative performance gap compared to a human evaluation of the Atomic test set, showing that the knowledge produced by the model approaches human performance. This result suggests that ℂ𝕆𝕄𝔼𝕋 could be effective with human evaluators in the loop to confirm the correctness of generated tuples.