The Second Conversational Intelligence Challenge (ConvAI2)
Table 8: Repeats in Mechanical Turk logs.
['[BOLD] Team Names', 'Engagingness', 'Unigram', 'Bigram', 'Trigram', 'Unique']
We then looked at how often the models repeated themselves in conversations with humans. ’s responses were unique in the logs. Again, it is clear the humans repeat themselves very infrequently, but there is not a clear relationship between our proxy measures of repetition with the human evaluation scores. We suspect this is because there are more subtle instances of repeating that our proxies do not measure, and the proxies have already been optimized by many models (e.g. by doing n-gram or full utterance blocking). For example we observed instances like “i like watching horror” followed by “i love watching scary movies” occurring, but these are not captured well by our metrics. Finally, overall utterance uniqueness should ideally be close to 100% with the same utterance rarely being repeated across conversations, with humans at 99%. While Hugging Face’s model was at 97%, many other models were lower, with the winner Lost in Conversation at 86%. A low uniqueness score could be problematic for a deployed system, as it might make users tire of it repeating itself. However, as our competition evaluations involve very short dialogues, this likely did not impact human evaluations.