How to Write Summaries with Patterns? Learning towards Abstractive Summarization through Prototype Editing
Table 4: Fluency and consistency comparison by human evaluation.
['[EMPTY]', 'Fluency', 'Consistency']
For the human evaluation, we asked annotators to rate each summary according to its consistency and fluency. The rating score ranges from 1 to 3, with 3 being the best. The kappa statistics are 0.33 and 0.29 for fluency and consistency respectively, and that indicates the moderate agreement between annotators. To prove the significance of these results, we also conduct the paired student t-test between our model and Re3Sum (row with shaded background). We obtain a p-value of 2\times 10^{-7} and 9\times 10^{-12} for fluency and consistency, respectively.