Dataset for Automatic Summarization of Russian News
Table 5: Human side-by-side evaluation
['Votes for winner', 'Reference wins', 'mBART wins']
There were no full draws, so we do not include them in the table. mBART wins in more than 73% cases. We cannot just conclude that it performs on a superhuman level from these results. We did not ask our annotators to evaluate the abstractiveness of the summaries. The reference summaries are usually too provocative and subjective. In contrast, mBART generates highly extractive summaries without any errors and with many essential details, and annotators tend to like it. The annotation task should be changed to evaluate the abstractiveness of the model. Even so, that is an excellent result for mBART.