Bootstrapping Generators from Noisy Data
Table 4: Rankings shown as proportions and mean ranks given to systems by human subjects.
['System', '1 [ITALIC] st', '2 [ITALIC] nd', '3 [ITALIC] rd', '4 [ITALIC] th', '5 [ITALIC] th', 'Rank']
Unsurprisingly, the human authored gold text is considered best (and ranked first 47% of the time). EDMTL is mostly ranked second and third best, followed closely by EDRL. The vanilla encoder-decoder system ED is mostly forth and Templ is fifth. As shown in the last column of the table (Rank), the ranking of EDMTL is overall slightly better than EDRL. We further converted the ranks to ratings on a scale of 1 to 5 (assigning ratings 5…1 to rank placements 1…5). This allowed us to perform Analysis of Variance (ANOVA) which revealed a reliable effect of system type. Post-hoc Tukey tests showed that all systems were significantly worse than RevAbs and significantly better than Templ (p < 0.05). EDMTL is not significantly better than EDRL but is significantly (p < 0.05) different from ED.