Better Rewards Yield Better Summaries: Learning to Summarise Without References
Table 1: Quality of reward metrics. G-Pre and G-Rec are the precision and recall rate of the “good” summaries identified by the metrics, resp. All metrics here require reference summaries. We perform stemming and stop words removal as preprosessing, as they help increase the correlation. For InferSent, the embeddings of the reference/system summaries are obtained by averaging the embeddings of the sentences therein.
['Metric', '[ITALIC] ρ', '[ITALIC] r', 'G-Pre', 'G-Rec']
From Table 1, we find that all metrics we consider have low correlation with the human judgement. More importantly, their G-Pre and G-Rec scores are all below .50, which means that more than half of the good summaries identified by the metrics are actually not good, and more than 50%