Controlling the Amount of Verbatim Copying in Abstractive Summarization
Table 3: The copy rate of various summarization models. We define copy rate as the percentage of summary n-grams appearing in the source text, where n=1/2/3/4 as well as an average of them. We experiment with selecting varying amounts of seen summary tokens ( ), unseen summary tokens ( ), and source tokens ( ) for training. A circle corresponds to about 5 million tokens for Gigaword and 385k tokens for Newsroom, which are used to compute the loss term.
['[EMPTY]', '[BOLD] Training Loss', 'Gigaword  [BOLD] 1-gram', 'Gigaword  [BOLD] 2-gram', 'Gigaword  [BOLD] 3-gram', 'Gigaword  [BOLD] 4-gram', 'Gigaword  [BOLD] Average', 'Gigaword  [BOLD] R-2', 'Newsroom  [BOLD] 1-gram', 'Newsroom  [BOLD] 2-gram', 'Newsroom  [BOLD] 3-gram', 'Newsroom  [BOLD] 4-gram', 'Newsroom  [BOLD] Average', 'Newsroom  [BOLD] R-2']
Copy rate is defined as the percentage of summary n-grams appearing in the source text. We set n=1/2/3/4 and the average of them. A high copy rate suggests that the summary is generated largely by copying verbatim from the source text. We experiment with selecting varying amounts of seen summary tokens ( ), unseen summary tokens ( ), and source tokens ( ) for training, where the number of circles is proportional to the number of tokens used in computing the loss term. Our findings suggest that, the factor that makes the most impact on the copying behavior of a summarizer is the proportion of seen and unseen summary words used for training the model. it only reuses source words during testing, despite that there is nothing to prevent the system from generating new words. The 1-gram copy rate for case a. is about 99% for both datasets, with the minor gap due to tokenization discrepancies. As more unseen words are used for training, the summarizer gradually transforms from copying only to both copying and generating new words not present in the source text. We observe that the ratio of seen vs. unseen words in ground-truth summaries is about 2:1 in both datasets, and Newsroom is slightly more extractive than Gigaword. Our analysis reveals that it is important to maintain a similar ratio during training in order to achieve high ROUGE scores. Pure extracts do not attain high ROUGE scores, as ground-truth summaries themselves are abstracts. Our analysis further suggests that training on source words has little impact on the copying behavior of the system, but it improves representation learning and has lead to consistently improved ROUGE-2 F-scores. Results show that our system achieves competitive performance, surpassing strong systems having reported results on this dataset, as judged by both metrics. These results demonstrate the effectiveness of our Transformer-based decoder-only architecture for abstractive summarization. Both BP-Norm and SBWR appear to be outstanding reranking methods, better than length normalization. Our observation also suggests that best-first search and beam search can produce similar outcome, despite that the two differ in their search strategies, with beam search visiting candidates according to summary length and best-first search favoring candidates having high log-likelihood scores. We suggest future work to explore other search methods such as A* search.