Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses
Table 2: BanditSum’s performance—calculated as the average between ROUGE-1,-2, and -L F1—on the validation set of the CNN/Daily Mail corpus. The sentence position information is perturbed at different levels, as explained in Section 4.
['train settingtest setting', 'original', 'random', 'reverse', 'insert-lead', 'insert-lead3', 'Mean', 'Std. Dev.']
when trained and tested on the various datasets. All models (except random) perform worse when tested on a mismatched data perturbation. Even when the distortion is at a single lead position in insert-lead and insert-lead3, the performance on the original data is significantly lower than when trained without the distortion. These results corroborate Kedzie et al. ’s findings for RL-based systems. Interestingly, the random model has the best mean performance and the lowest variation indicating that completely removing the position bias may allow a model to focus on learning robust sentence semantics.