A Dataset for Document Grounded Conversations
Table 5: The distribution of BLEU score for conversations with more than 10 turns.
['Percentile', '20', '40', '60', '80', '99']
Split Criteria : We automatically measure the quality of the conversations using BLEU Papineni et al. We use BLEU because we want to measure the overlap of the turns of the conversation with the sections of the document. Hence, a good quality conversation should use more information from the document than a low quality conversation. We divide our dataset into three ratings based on this measure. The BLEU score is calculated between all the utterances {x1,…,xn} of a conversation Ci and the document di corresponding to Ci. We eliminate incomplete conversations that have less than 10 turns. We split the dataset into three ratings based on BLEU score.