Deep Reinforced Self-Attention Masks for Abstractive Summarization (DR.SAS)
Table 1: ROUGE Metrics for Baseline, Fine-tuned UniLM and Our Method (Finetuning UniLM in sync with AC)
['Metrics (F1 score)', 'Baseline', 'UniLM', 'AC + UniLM']
In order to evaluate our model, we conducted visual and quantitative analyses. Quantitatively, we evaluate our method in terms of ROUGE. ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is a set of metrics for evaluating automatic summarization of texts as well as machine translation. It works by n-gram comparison between a generated and and reference summary. We compared our model to extractive summarization and the abstractive summarization by UniLM as our baselines. The extractive method selects sentences from a given document to construct a summary of the passage. Specifically, we adapt TextRank The baseline can be thought of as a process of recommendation where one sentence recommends another sentence to which it is similar. The baseline was evaluated on the CNN/DM test set. The oracle is essentially a method that can generate the ground truth summaries and would have a ROUGE score of 1. The marginal improvement can be attributed to the bias of the reinforced actor-critic policy gradient towards better ROUGE (AC - Reward).