CohEval: Benchmarking Coherence Models
Table 6: Agreement shows the AC1 agreement for the pairwise ranking of the generated summaries between two annotators and the models. The last four columns show the average rankings (over the test set) assigned by the models to the different summarization systems.
['[BOLD] Models', '[BOLD] Agr.', '[BOLD] Summarization Systems  [BOLD] BSEA', '[BOLD] Summarization Systems  [BOLD] UniLM', '[BOLD] Summarization Systems  [BOLD] PG', '[BOLD] Summarization Systems  [BOLD] SENE']
For the user study, the agreement between the two annotators was 0.78, which indicates fairly reliable data. EGrid and LexNeuralEGrid show the highest agreement with human judgements. This coherence-based evaluation of the summarization systems can be considered a real-world use-case for the coherence models. Each summary is assigned a rank between 1 and 4 based on the coherence scores produced by the models. The average ranking is computed by averaging the rankings over all the test samples. It can been seen that the coherence models with higher human agreement, EGrid and LexNeuralEGrid yield similar rankings; they rank the summaries produced by UniLM as more coherent than others in general.