Neural Summarization by Extracting Sentences and Words
Table 2: Rankings (shown as proportions) and mean ranks given to systems by human participants (lower is better).
['Models', '1 [ITALIC] st', '2 [ITALIC] nd', '3 [ITALIC] rd', '4 [ITALIC] th', '5 [ITALIC] th', '6 [ITALIC] th', 'MeanR']
Specifically, we show, proportionally, how often our participants ranked each system 1st, 2nd, and so on. Perhaps unsurprisingly, the human-written descriptions were considered best and ranked 1st 27% of the time, however closely followed by our nn-se model which was ranked 1st 22% of the time. The ilp system was mostly ranked in 2nd place (38% of the time). The rest of the systems occupied lower ranks. We further converted the ranks to ratings on a scale of 1 to 6 (assigning ratings 6…1 to rank placements 1…6). This allowed us to perform Analysis of Variance (ANOVA) which revealed a reliable effect of system type. Specifically, post-hoc Tukey tests showed that nn-se and ilp are significantly (p<0.01) better than lead, nn-we, and nn-abs but do not differ significantly from each other or the human goldstandard.