This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation
Table 5: Human evaluation. ∗ indicates the difference from our system is statistically significant with p<0.01 under a paired t-test.
['[EMPTY]', 'Overall', 'Informative', 'Fluent']
For overall scores, the baselines are all between 1.5 and 2.0, indicating the subjects are usually considered as poor or fair (recall that the scale is 1-4, with 4 being the highest). Our system is 2.28, while the original subject and human annotation are between 2.5 and 3.0. This means more than half of our system outputs are at least fair, and the original subject and human annotation are often good or great. We also find that in 89 out of 500 emails, our system outputs have ratings higher than or equal to the original and human annotated subjects. Furthermore, the raters prefer the human annotated subject to the original subject.