CohEval: Benchmarking Coherence Models
Table 9: Utterance ranking results for different coherence models on Advising and Ubuntu datasets. R@k indicates Recall@k, X indicates result not shared.
['[EMPTY]', '[BOLD] R@1', '[BOLD] R@5', '[BOLD] R@10', '[BOLD] MRR', '[BOLD] Acc.']
The DSTC8 challenge ranking considers the average of Recall@1, Recall@5, Recall@10 and Mean Reciprocal Rank (MRR). We report both the official evaluation results and the coherence models’ performance even though the latter is tested on the refined datasets. From the results, we see that the overall performance of all the coherence models is quite poor. This is not particularly surprising, since this is a considerably harder task for the coherence models alone. However, given that some models perform relatively well (e.g., NeuralEGrid on the Ubuntu dataset), the insights of these models can be incorporated into the dialogue systems to make them sensitive to the context.