Real-Time Web Scale Event Summarization Using Sequential Decision Making
Figure 4: Average system performance. LsFs and LsCosFs runs are trained and evaluated on first sentences only (like the Cos system). Unpenalized results are omitted for space but the rankings are consistent.
['[EMPTY]', 'latency-penalized exp. gain', 'latency-penalized comp.', 'latency-penalized  [ITALIC] F1']
Since Cos only considers the first sentence of each document, it may miss relevant sentences below the article’s lead. In order to confirm the importance of modeling the oracle, we also trained and evaluated the Ls based approaches on first sentence only streams. only runs. The Ls approaches still dominate Cos and receive larger positive effects from the latency penalty despite also being restricted to the first sentence. Clearly having a model (beyond similarity) of what to select is helpful. Ultimately we do much better when we can look at the whole document.