Simple Unsupervised Keyphrase Extraction using Sentence Embeddings
Table 2: Comparison of our method with state of the art on the three datasets. Precision (P), Recall (R), and F-score (F1) at 5, 10, 15 are reported. Two variations of EmbedRank with λ=1 are presented: s2v uses Sent2Vec embeddings, while d2v uses Doc2Vec.
['N', 'Method', 'Inspec P', 'Inspec R', 'Inspec F1', 'DUC P', 'DUC R', 'DUC F1', 'NUS P', 'NUS R', 'NUS F1']
In the context of typical Web-oriented use cases, most data comes as either very short documents (e.g. tweets) or medium ones (e.g. news articles). The expected performance for Web applications is thus closer to the one observed on the Inspec and DUC2001 datasets, rather than on NUS. In this section, we add EmbedRank++ to the evaluation using the same three datasets. If the result is confirmed by future tests, it casts a shadow on using solely F-score as an evaluation measure for keyphrase quality. A similar issue was shown to be present in Information Retrieval test collections Tonon et al. We acknowledge that the presented study is a preliminary one and does not support a strong claim about the usefulness of the F-score for the given problem. It does however show that people dislike redundancy in summaries and that the λ<1 parameter in EmbedRank is a promising way of reducing it.