Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
Table 5: Human judgments for Plausibility of the different models. Each numerical cell contains a percentage value corresponding to its row truncated to 2 decimal precision.
['Model Alias', 'No(%)', 'Unsure(%)', 'Yes(%)']
We conducted a survey on the crowd-sourcing platform, Amazon Mechanical Turk. Every model response is scored on 2 categories: 1) Plausibility - is the response plausible for the given source? and 2) Content Richness - does the response add new information to the conversation? We asked the evaluators to respond on a 5-point scale to the questions above (Strongly Agree, Agree, Unsure, Disagree, Strongly Disagree). These were later collapsed to 3 categories (Agree, Unsure, Disagree). We computed p-values for each pair of models: MMI200 vs DC-MMI200 and TA vs DC-MMI200. For plausibility, we did not find a significant difference in either comparison (p-value ≈0.25) while for content richness, both differences were found to be significant (p-value <10−4). To summarize: our model significantly beats both baselines in terms of content richness while the difference in plausibility was not found to be statistically significant.