Scientific document summarization via citation contextualization and scientific discourse {}^{*} This is a pre-print of an article published on IJDL. The final publication is available at Springer via http://dx.doi.org/10.1007/s00799-017-0216-8
Table 11: Effect of learning algorithms in identifying the discourse facets. SVM: Support Vector Machine with Linear Kernel, RF: Random Forest, LR: Logistic Regression, Oracle: Highest achievable score. Numbers are weighted accuracy scores by annotators.
['[EMPTY]', 'SVM', 'RF', 'LR', 'Oracle']
Since for the TAC dataset there are 4 annotators, and the official metric is weighted accuracy scores, we also calculate the oracle score by always predicting what the majority of the annotators agree on. The oracle achieves 0.67 percent, suggesting that identifying discourse facets is not trivial for humans. We can see that the SVM classifier achieves the highest results with 81% relative accuracy to the oracle. For the CL-SciSum dataset, there is only one annotator per discourse facet and therefore, the weighted accuracy metrics translates to simple accuracy scores.