CohEval: Benchmarking Coherence Models
Table 7: AC1 Agreement for the pairwise ranking of the generated summaries between human and the models. The last five columns show the average rankings.
['[BOLD] Models', '[BOLD] Agr.', '[BOLD] Summarization Systems  [BOLD] S-6', '[BOLD] Summarization Systems  [BOLD] S-13', '[BOLD] Summarization Systems  [BOLD] S-26', '[BOLD] Summarization Systems  [BOLD] S-16', '[BOLD] Summarization Systems  [BOLD] S-18']
We show the average rankings for different extractive systems (e.g., S-6, S-13, …) in the last five columns. NeuralEGrid has a higher agreement with human judgement compared to the other models. It also yields an average ranking very similar to the human average ranking. By showing that models with high human agreements can produce reasonable rankings (see agreements) of summarization systems, we demonstrate that evaluating summarization systems using coherence as a measure is viable as a research objective. We hope that this can serve to motivate summarization evaluation towards the coherence of the summaries rather than purely focusing on the improvements in ROUGE scores.