Measuring academic influence: Not all citations are equal
TABLE 3: Additive improvement of model performance. We report macro-averages under ten-fold cross-validation. With respect to the F-measure, asterisks (*) mark models that are better than model (2) whereas daggers (†) mark models that are better than model (3).
['Model (1)', 'Features random', 'F-measure 0.10', 'F-measure', 'Precision 0.10', 'Recall 0.10']
We included two baselines. The first baseline randomly labels a reference with a probability equal to the distribution of the labels in the training data. The second predicts the academic influence of references based on their Google citation counts (aux_citeCount). Note that the macro-averaged F-measure is not necessarily between the averaged precision and recall. For example, for model (3) in the table, 0.35 is not between 0.36 and 0.41. The models marked with an asterisk (*) are statistically significantly better than the two baselines. We use a one-tailed paired t-test with a 99% significance level. The model uses only four features, two of which are count-based (countsInPaper) and one semantics-based (sim_titleCore). Adding more features to model (6) did not result in further improvement. This observation supports the hypothesis that feature selection is useful for this task. In general, feature selection removes useless or detrimental features, which often leads to better performance (e.g., higher F-measures) and greater efficiency. With logistic regression, the magnitude (absolute value) of the weights, βi, indicates the importance of the corresponding feature in the model. We used the mnrfit command in Matlab to conduct logistic regression on our data. The weights assigned to the features countsInPaper_whole, sim_titleCore, countsInPaper_secNum, and aux_selfCite are 2.7228, 1.2683, 1.1763, and -0.0923. the least important is aux_selfCite. Note that the weight for aux_selfCite is smallest, which corresponds to the observation that self-citations are less likely to be influential. We have also used logistic regression to classify the references, using the same experimental setup as we used for the SVM. The logistic regression performance is slightly below that of the SVM (≈0.37 vs. 0.41).10endnote: 10With logistic regression, the precision and recall are 0.38 and 0.42. This curve serves as a minimum baseline for comparison with the other features. Model (3) was trained and tested with ten-fold cross validation, whereas the F-measure of 0.37 is based on using the whole dataset as training data, with no independent testing data; thus the small gap in the F-measures (0.35 versus 0.37) indicates that SVM is performing well.