Generating Representative Headlines for News Stories
Table 4. Robustness Comparison of Different Attention Designs on the Noisy Dataset.
['[BOLD] Method', 'R1-P', 'R1-R', '[BOLD] R1-F', 'R2-P', 'R2-R', '[BOLD] R2-F', 'RL-P', 'RL-R', '[BOLD] RL-F', '[BOLD] Len-C', '[BOLD] Len-W']
Among all attention modules, Referee Attention achieves the worst performance, which is as expected since Referee Attention can assign inappropriately large weights to noisy articles, severely corrupting the final output. For comparison, the simple Uniform Attention module will at least avoid focusing on the wrong articles, and hence achieves better performance than Referee. From the model perspective, the Referee model is more complex and is harder to train with limited resources. This experiment also shows that, when on-topic articles dominate the story, even simple uniform attention can achieve satisfactory performance by integrating decoding outputs from different articles, but the traditional Referee Attention can produce risky results. Our Self-voting-based Attention achieves best performances under all settings, thanks to its capability of leveraging the dynamic voting process between articles to emphasize shared common information and identify noises. When tested without any pre-training (H), where the initial article representations are far from perfect, the performance gap between different attention designs becomes more vivid.