Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
Table 4: Automatic metrics evaluation. The 3rd and 4th columns show the ratio of types to tokens for unigrams and bigrams respectively. 7th Column shows the % of stop-words generated by the models in their responses.
['Model', 'Alias', 'distinct-1', 'distinct-2', 'BLEU -1', 'Avg. length', 'Stop-word%']
Following Li et. al. Li et al. These are the ratios of types to tokens for unigrams and bigrams, respectively. We also report BLEU-1 scores following previous work, however it should be noted that BLEU-1 is not generally accepted to correlate with human judgments in conversation generation tasks Liu et al. Lastly, we compare the percentage of stop-words The human responses are diverse and also generally longer than automatically generated responses. MMI200 has higher diversity than TA-Seq2Seq in terms of distinct-1 and distinct-2. This illustrates the importance of re-ranking using MMI. Our approach produces almost twice as many distinct unigrams and bigrams. We also observe MMI200 and TA-Seq2Seq achieve higher BLEU scores than our models, however this is not surprising since our models are designed to generate more interesting responses containing rarer content words that are less likely to appear in reference responses. As expected we observe that MMI200 and TA-10 have a higher percentage of stop-words than human responses.