Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised
Table 5: Summarization results on OpoSum.
['[BOLD] Summarization Random', 'ROUGE-1 ROUGE-2 ROUGE-L 35.1', 'ROUGE-1 ROUGE-2 ROUGE-L 11.3', 'ROUGE-1 ROUGE-2 ROUGE-L 34.3']
Opinion Summaries We produced opinion summaries for the 60 products in our benchmark using a two-stage procedure. First, all reviews for a product were shown to three annotators. Each annotator read the reviews one-by-one and selected the subset of segments they thought best captured the most important and useful comments, without taking redundancy into account. This phase produced binary salience labels against which we can judge the ability of a system to identify important opinions. Again, using the Kappa coefficient, agreement among annotators was K=0.51 (N=8,175, k=3). In the second stage, annotators were shown the salient segments they identified (for every product) and asked to create a final extractive summary by choosing opinions based on their popularity, fluency and clarity, while avoiding redundancy and staying under a budget of 100 words. For every product, we treated one reference summary as system output and computed how it agrees with the rest. Assisting Opinosis with aspect predictions is beneficial, however, it remains significantly inferior to our model (see the supplementary material for additional results).