A Generative Approach to Titling and Clustering Wikipedia Sections
Table 3: Rouge-L on Wikipedia section heading generation by length. The attention-based decoders outperform the decoders without attention on target texts containing 5-10 tokens, but not on shorter target sequences.
['# Tokens', '1-5', '5-10', '10-15', '15+']
Because the Wikipedia dataset contains diverse types of headings, including short abstractive headings and long extractive headings, we subdivide our test data in order to better understand model performance. The Rnn and Rnn+Sc models outperform the attention-based models on data with low extractive scores (≤0.5). The attention-based models are not better at producing long titles in general, but rather their ability to copy from the input text allows them to generate long titles when they are extractive. We can further examine this trend by computing correlations between Rouge-L and extractive score. The idea behind a partial correlation is to identify the relationship between two variables X and Y that is not explained by a confound Z. We first compute the residuals eX,i and eY,i, and then compute the correlation between these residuals: