SemRe-Rank: Improving Automatic Term Extraction By Incorporating Semantic Relatedness With Personalised PageRank
Table 2. Number of candidate terms extracted by each ATE method on each dataset and their maximum Recoverable True Positives (RTP). The voting method is not included as it uses the output (i.e., same set of candidate terms) from other ATE methods. We use publicly available implementations of these methods and due to the difference in such implementations, it has been impossible to ensure they use identical linguistic filters and extract the identical set of candidate terms. See Section 5.3.1 for acronyms of base ATE methods.
['Dataset', 'Ground truth', 'ATE methods: Basic, ComboBasic, LP, NTM, PU Candidate terms', 'ATE methods: Basic, ComboBasic, LP, NTM, PU RTP', 'ATE methods: TFIDF, CValue, RAKE, Weirdness, Relevance, GlossEx,  [ITALIC] χ2 Candidate terms', 'ATE methods: TFIDF, CValue, RAKE, Weirdness, Relevance, GlossEx,  [ITALIC] χ2 RTP']
The second measure is inspired by the ‘R-Precision’ used in information retrieval, that is the Precision at the Rth position in the ranking of results for a query that is expected to have R relevant documents. In this work we propose to calculate Precision (P), Recall (R, number of true positives as a fraction of the number of ground truth), and F1 (harmonic mean of P and R) at a K that equals to the size of the intersection of the extracted candidate terms and the ground truth. In other words, this is the number of expected real terms in the candidates, and we refer to this as the number of ‘Recoverable True Positives’, or RTP. Note that the RTPs of an ATE method may only be a subset of the ground truth for a dataset since no linguistic filters are guaranteed to cover all lexical and syntactic patterns of terms. Also, different ATE methods can use different linguistic filters and therefore, for the same dataset, different ATE methods extract different candidate terms and can have different RTP values. Using the GENIA dataset as an example, we calculate P, R, F1 at rank K=13,831 for the Basic method, and K=15,603 for the CValue method. Intuitively, a perfect ATE method will obtain 100% precision and also maximum obtainable recall on that dataset at rank K=RTP. We will refer to this measure as Precision, Recall and F1 at K=RTP, or in short, P@RTP, R@RTP, and F1@RTP (also the F1 mentioned in the abstract and introduction of this article). For example, for the Basic method on the TTCw dataset, the RTP is just over 200 and the candidate terms extracted are over 59,000. In other words, we expect the method to rank just over 200 real terms highly out of over 59,000 candidates. This is a much more challenging task than, e.g., on the GENIA dataset which has over 13,000 RPT’s and over 56,000 candidate terms for the same ATE method. Also, effectively this means that for TTCm and TTCw, the maximum attainable P@K for K >RTP will be significantly lower. For example, at K=2,000 for TTCm, the maximum attainable precision by this method is only 11% (0.11) (2262,000).