The Second Conversational Intelligence Challenge (ConvAI2)
Table 7: Rare word frequencies in Mechanical Turk logs.
['[BOLD] Team Names', 'Engagingness (1-4)', 'Freq1h (model)', 'Freq1h (human)', 'Freq1k (model)', 'Freq1k (human)']
Rare Word Statistics We also looked to see how often rare words were used in the conversation logs. The hypothesis here is that utterances with some rare words might be less generic and hence more interesting/engaging, rendering higher human evaluation scores. The results show that humans use significantly more rare words than any of the models, and the bottom three models do have lower Freq1h scores than the top three; otherwise, however, the relationship between evaluation score of the models and their use of rare words is not completely clear. We suspect that is because this is just one factor among many that would need to be disentangled.