Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
Table 4: Fleiss’ Kappa of Human Agreement. Two testing scenarios Appropriate (App.) and Informativeness (Inf.) are used to evaluate the the quality of generated response. The Fleiss’ Kappa evaluates agreement from various annotators and focuses on the comparison of two models with three categories: win, tie and loss.
['Model', 'App.', 'Inf.']
For each case, the response from ConceptFlow is compared to the responses from two baseline models, CCM and GPT-2 (conv). The comparison result is divided into three categories: win, tie and loss. Then the human evaluation agreement is calculated with Fleiss’ Kappa (κ). The κ value ranges from 0.21 to 0.40 indicating fair agreement, which confirms the quality of human evaluation.