Multi-stage Pretraining for Abstractive Summarization
Table 4: Label coverage of our BERT-based content selection model. We also show the performance of a content selection oracle which always does perfect content selection. True positives here are groundtruth word pieces that are selected by the content selector. The oracle achieves perfect precision because the labels are used to select inputs in the first place. The oracle does not achieve perfect recall because not all input word pieces are present in the groundtruth.
['CNN/DM dev', 'Oracle', 'Precision 100.00', 'Recall 80.77', 'F1 89.01']
Label coverage is an important metric for understanding the performance of content selectors in the context of copy mechanisms. If the content selector has a false negative on a label word, then that word cannot be copied, thus hurting performance. Similarly, if the content selector has a false positive on a label word, the usefulness of the content selector model degrades. Our content selection model seems sub-par compared to the oracle, with slightly less than half of the labels present in the content selector’s outputs, and slightly more than half of the content selector’s outputs present in the groundtruth labels.