Combining Lexical and Syntactic Features for Detecting Content-dense Texts in NewsIn submission to JAIR
Table 8: Binary classification accuracies(%) on basic human annotated datasets for models trained on heuristically labeled data.
['[BOLD] Business', 'Combined', 'Agreed', 'Adjudicated', 'Anno_1', 'Anno_2']
The second baseline (Baseline-2) is established based on the length of the entire news article, not only the lead. The intuition is that longer articles may have uninformative leads designed to draw the reader into the subject while short articles need to start out with a more focused presentation of the event so are likely to have an content-dense lead. We train a L2-regularized logistic regression model based on this single feature. Accuracy computed against each of the two individual annotators is shown in the last two columns. Sports and politics domains have higher prediction accuracies on the data labeled by the first annotator, and business and science domains have higher prediction accuracies for the second annotator’s labels. Also the prediction accuracies have smaller variances on the data labeled by the first annotator, between 78% for the politics domain and 74% for science the domain, compared with the accuracies on data labeled by the second annotator, between 87% for business and 71% for sports. Overall however the prediction accuracy on the final combined data, after disagreements have been adjudicated, is highest, demonstrating that the adjudication procedure did lead to more internally consistent labels. As in the heuristically labeled data, recognition accuracies are higher for the business and science domains (83%) and lower for the sports and politics domains (around 80%).