Combining Lexical and Syntactic Features for Detecting Content-dense Texts in NewsIn submission to JAIR
Table 5: Inter-annotator agreement on manual annotations. Percent agreement is computed on the binary annotation, correlation is computed on the real-value degree of content-density of the leads. All correlations are highly significant, with p<0.001.
['[EMPTY]', 'Agreement', 'Kappa', 'Correlation']
For the binary annotations we also report the Kappa statistic. Agreement is highest—almost 80%— for the politics domain. The agreement is lowest in the business domain, 70%. The correlations of content-density scores exceed 0.5 and are highly significant (p<0.001) for all domains. The high correlations of real-valued scores, especially for the politics and business domains, suggest that the task may be more amenable to annotation and automation as a real-value prediction task rather than as a binary distinction.