Generating an Overview Report over Many Documents
Table 4: Overall performance
['[BOLD] rank', '[BOLD] model', '[BOLD] human eval.', '[BOLD] time', '[BOLD] coverage', '[BOLD] diversity']
Let the three reports for the same corpus be the three alternatives, denoted by a1,a2,a3. Let the human evaluation mean score, running time, information coverage score, and topics diversity score be four criteria, denoted by c1,c2,c3,c4. Next, we use Saaty’s pairwise comparison 9-point scale to determine weights for each criterion. A weight vector w={w1,w2,w3,w4} is then computed using the Analytic Hierarchy Process (AHP) procedure, where wi is the weight for criterion ci. A weighted normalization decision matrix T is then generated from the normalized matrix R and We can see that the overview report generated on 0.2-summaries achieves the best overall performance on both BBC News and Factiva-Marx.