Toward Abstractive Summarization Using Semantic Representations
Table 2: Percentage of summary edges that can be covered by an automatically constructed source graph.
['[EMPTY]', '[BOLD] Summary Edge Coverage (%)', '[BOLD] Summary Edge Coverage (%)', '[BOLD] Summary Edge Coverage (%) Expand', '[BOLD] Summary Edge Coverage (%) Expand']
We investigated how well these automatically constructed source graphs cover the gold-standard summary graphs produced by AMR annotators. ‘Unlabeled’ counts edges as matching if both the source and destination concepts have identical labels, but ignores the edge label. In order to improve edge coverage, we explore expanding the source graph by adding every possible edge between every pair of concepts within the same sentence. We also explored adding every possible edge between every pair of concepts in the entire source graph. Edges that are newly introduced during expansion receive a default label ‘null’. Subgraph prediction became infeasable with the document-level expansion, so we conducted our experiments using only sentence-level expansion. Sentence-level graph expansion increases the average number of edges by a factor of 15, to 3,292. Fig. Document-level expansion covers the gold-standard summary edge “chase-01” → “garden,” yet the expansion is computationally prohibitive; sentence-level expansion adds an edge “dog” → “garden,” which enables the prediction of a structure with similar semantic meaning: “ Joe’s dog was in the garden chasing a cat.” Oracle results for the subgraph prediction stage are obtained using the ILP decoder to minimize the cost of the output graph, given the gold-standard. We assign wrong nodes and edges a score of −1, correct nodes and edges a score of 0, then decode with the same structural constraints as in subgraph prediction. The resulting graph is the best summary graph in the hypothesis space of our model, and provides an upper bound on performance achievable within our framework. Oracle performance on node prediction is in the range of 80% when using gold-standard AMR annotations, and 70% when using JAMR output. Edge prediction has lower performance, yielding 52.2% for gold-standard and 31.1% for JAMR parses. When graph expansion was applied, the numbers increased to 64% and 46.7%, respectively.