The YLI-MED Corpus: Characteristics, Procedures, and Plans
Table 19: Comparison of annotator agreement and confidence in positive judgments for Training vs. Test sets.
['Set (Positives)', '% Partial Agreement', 'Avg. Confidence for Partial', '% Full Agreement', 'Avg. Confidence for Full', 'Total Avg. Confidence']
It also shows that the differences in agreement are not statistically significant, nor are the differences in average confidence.