The Second Conversational Intelligence Challenge (ConvAI2)
Table 5: Human Evaluation Results
['[BOLD] Team Names', 'Engagingness (1-4)', 'Persona Detection (0-1)']
Lost in Conversation won the competition with an engagingness score of 3.11 out of 4. The calibration did not affect the ordering of the scores, and the scores reported in the final leaderboard are post-calibration. Recall that this score is the percentage of the time that the annotaters were able to to distinguish the model’s persona from a random one. We often observed models repeating the persona sentences almost verbatim, which might lead to a high persona detection score but a low engagingness score. Training models to use the persona to create engaging responses rather than simply copying it remains an open problem.