Facet-Aware Evaluation for Extractive Summarization
Table 7: Correlation between ground-truth and estimated FAR scores by Pearson’s r, Spearman’s ρ, and Kendall’s τ. N denotes the number of support groups.
['[BOLD] Method', '[ITALIC] r', '[ITALIC] N=1  [ITALIC] ρ', '[ITALIC] τ', '[ITALIC] r', '[ITALIC] N=2  [ITALIC] ρ', '[ITALIC] τ', '[ITALIC] r', '[ITALIC] N=3  [ITALIC] ρ', '[ITALIC] τ']
Correlation w. Human-Annotated FAMs. We then explore the correlation between human-annotated and machine-created FAMs by evaluating extractive methods against both of them. This time we extend to find for each facet multiple support sentences and put each support sentence into a separate support group. We measure the correlation between estimated and ground-truth FAR by Pearson’s r. We measure the correlation between system rankings induced from estimated and ground-truth FAR by Spearman’s ρ and Kendall’s τ. We observe that creating three support groups consistently shows the highest correlation for the same sentence regression approach. Also, the FAMs created by ROUGE-1 F1 and ROUGE-AVG F1 have very high correlation with human annotation, indicating the usability and reliability of machine-created FAMs for system ranking.