Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Table 2: Automatic Evaluation results (higher is better for all metrics). Our best models (CMR+w and CMR) considerably increase the quantitative measures of Grounding, and also slightly improve Diversity. Automatic measures of Quality (e.g., Bleu-4) give mixed results, but this is reflective of the fact that we did not aim to improve response relevance with respect to the context, but instead its level of grounding. The human evaluation results in Table 3 indeed suggest that our best system (CMR+w) is better.
['[EMPTY]', '[BOLD] Appropriateness Nist', '[BOLD] Appropriateness Bleu', '[BOLD] Appropriateness Meteor', '[BOLD] Grounding Precision', '[BOLD] Grounding Recall', '[BOLD] Grounding F1', '[BOLD] Diversity Entropy-4', '[BOLD] Diversity Distinct-1', '[BOLD] Diversity Distinct-2', 'Len']
We measure the overall relevance of the generated responses given the conversational history by using standard Machine Translation (MT) metrics, comparing generated outputs to ground-truth responses. The latter metric is a variant of Bleu that weights n-gram matches by their information gain by effectively penalizing uninformative n-grams (such as “I don’t know”), which makes it a relevant metric for evaluating systems aiming diverse and informative responses. MT metrics may not be particularly adequate for our task Liu et al. In terms of appropriateness, the different variants of our models outperform the Seq2Seq and MemNet baselines, but differences are relatively small and, in case of one of the metrics (Nist), the best system does not use grounding. Our goal, we would note, is not to specifically improve response appropriateness, as many responses that completely ignore the document (e.g., I don’t know) might be perfectly appropriate. Our systems fare much better in terms of Grounding and Diversity: our best system (CMR+w) achieves an F1 score that is more than three times (0.38% vs. 0.12%) higher than the most competitive non-MRC system (MemNet).