The Second Conversational Intelligence Challenge (ConvAI2)
Table 3: Automatic Metrics Leaderboard.
['[BOLD] Team Names', 'Perplexity', 'Hits@1', 'F1']
We had over 23 teams submit models to be evaluated for the automatic metrics. The rank of each team was determined by sorting by the minimum rank of the score in any of the three metrics (F1, Hits@1, and Perplexity). The Hugging Face team performed the best in every single metric and was therefore determined to be the winner of this round. Retrieval models fared a little worse than generative models in the human evaluations, although we are unsure if this is true in general, or because no very strong retrieval model was proposed. Hugging Face used a two-head Transformer model, but opted to generate rather than retrieve). In our opinion, looking at the outputs from the generative systems in the competition, they still fall short of the most interesting and engaging comments of humans (which sometimes retrieval models choose); however, the generic responses from generative models are often low-risk or “safe” responses, which may give them higher scores.