The Second Conversational Intelligence Challenge (ConvAI2)
Table 9: Blind Evaluation Results. Average engagingness score (1-4) for the randomly sampled subset of conversations.
['[EMPTY]', '[BOLD] Hugging Face', '[BOLD] Lost in Conversation']
To better understand this, we performed a blind evaluation ourselves of a random sample of the Mechanical Turk evaluation logs from these two teams, giving each conversation a score between 1 and 4 and making comments about the model’s performance. As you can see, despite the apparent annotator bias, each annotator agreed with the Turkers regarding which model was better.