Examining the State-of-the-Art in News Timeline Summarization
Table 5: Proportion of adjacent dates of timelines produced by different methods, and the ground-truth timelines.
['[EMPTY]', 'Ground-truth', 'Martschat2018', 'Datewise', 'Clust']
Automatically constructed timelines often contain a high amount of multiple adjacent dates, while this is not the case in ground-truth timelines. Summaries of such adjacent dates often tend to refer to the same event and introduce redundancy into a timeline. To quantify this, we count the proportion of those ”date bigrams” in a chronologically ordered timeline, which are only 1 day apart. The results (see Note that Martschat2018 includes an objective function to reward diversity within a timeline, while Datewise has no explicit mechanism against redundancy among separate dates. Interestingly, when forcing Datewise to avoid selecting adjacent dates (by skipping such dates in the ranked list), the performance in all metrics decreases. In this case, high redundancy is a safer strategy for optimizing TLS metrics compared to enforcing a more balanced spread over time. Because of such effects, we advise to use automated evaluation metrics for TLS with care and to conduct qualitative analysis and user studies where possible.