Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access
Table 6: Human evaluation results on knowledge-grounded response generation
['Method', 'Baseline', 'Appropriateness %W', 'Appropriateness %L', 'Appropriateness %Tie', 'Accuracy %W', 'Accuracy %L', 'Accuracy %Tie']
In addition, we also performed human evaluations of the generated responses with the following two crowdsourcing tasks: Appropriateness: given a dialogue context and a pair of responses generated by two methods, crowdworkers were asked to select a more appropriate response to the context. Accuracy: given a knowledge snippet and a pair of responses generated by two methods, crowdworkers were asked to select a more accurate response to the knowledge. In both tasks, we presented each instance to three crowdworkers; asked them to choose either response or ‘not sure’ for the cases that are equally good or bad; and took the majority as the final label for the instance. Comparing between two GPT-2 variants, the model with knowledge provided more accurate information based on explicitly given knowledge than the one without knowledge. However, this accuracy gap between two models is not very big, which depicts the need to add more diversity in knowledge content which cannot be handled just by memorizing facts from the training data.