Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses
Table 4: Average ROUGE-1, -2 and -L F1 scores on Dearly, and Dmed, Dlate. Each set contains 100 documents.
['Model', '[ITALIC] Dearly', '[ITALIC] Dmed', '[ITALIC] Dlate']
We also examined the auxiliary loss model on documents where the summary is mostly comprised of lead sentences Dearly, mostly sentences much later in the article Dlate, and a dataset at the midway point, Dmed. To create these sets, we rank test articles using the average index of its summary sentences in the source document. The 100 test articles with lowest average index are Dearly, the 100 with highest value are Dlate and the 100 closest to the median are Dmed. While all systems are quite close to the oracle on Dearly they only reach half the performance on Dlate. This gap indicates that our improvements only scratch the surface, but also that this problem is worthy and challenging to explore.