This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation
Table 6: Correlation analysis between the automatic scores and the human evaluation.
['[EMPTY]', 'Pearson’s  [ITALIC] r', 'Spearman’s  [ITALIC] ρ']
It is important to check if the automatic metric scores can truly reflect the generation quality and serve as valid metrics for subject line generation. To this end, we take the average of three human ratings and then calculate Pearson’s r and Spearman’s ρ between different automatic scores and the average human rating. We also report the inter-rater agreement in the last row by checking the correlation between the third human rating and the average of the other two. We find that the inter-rater agreement is moderate with 0.64 for Pearson’s r and 0.58 for Spearman’s ρ. We would recommend ESQE because it has the highest correlations while being reference-less.