Improving Abstraction in Text Summarization
Table 3: Ablation study on the validation set of the anonymized CNN/Daily Mail dataset.
['[BOLD] Model', '[BOLD] R-1', '[BOLD] R-2', '[BOLD] R-L', '[BOLD] NN-1', '[BOLD] NN-2', '[BOLD] NN-3', '[BOLD] NN-4']
In order to evaluate the relative impact of each of our individual contributions, we run ablation studies comparing our model ablations against each other and against the novelty baseline. Results show that our base model trained with the maximum-likelihood loss only and using the language model in the decoder (ML, with LM) has higher ROUGE scores, novel unigrams, and novel bigrams scores than our base model without the language model (ML). ML with LM also beats the novelty baseline for these metrics. When training these models with reinforcement learning using the ROUGE reward (ML+RL ROUGE and ML+RL ROUGE with LM), the model with language model obtains higher ROUGE-1 and ROUGE-2 scores. However, it also loses its novel unigrams and bigrams advantage. Finally, using the mixed ROUGE and novelty rewards (ML+RL ROUGE+Novel) produces both higher ROUGE scores and more novel unigrams with the language model than without it. This indicates that the combination of the language model in the decoder and the novelty reward during training makes our model produce more novel unigrams while maintaining high ROUGE scores.