Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression
Table 4: Human evaluation results on the Gigaword dataset. Best results are boldfaced. Note that we are directly comparing with human-generated compressions. All differences (within same column) are stat. significant with p<0.05.
['[EMPTY]', 'Readability', 'Informativeness']
We can see that for this dataset our F1 is still far away from state-of-the-art. We reason that this is partly because the ground-truth compressions in Google dataset are based on news headlines (later edited by human) rather than compressions generated directly by human beings. This is expected because our model is syntax-aware rather than semantic-aware. For example, the Deleter would readily delete negation words (e.g., from ” i do not love NLP” to ”i do love NLP”), which usually causes abrupt change in meaning. We leave enforcing semantic consistency by leveraging Neural Language Inference datasets (williams2017broad) as future work. It is also interesting to note that there is a larger gap between Annotator #1 and #2 than between Annotator #1 and our model, indicating that the quality of compression is a fairly subject matter.