The YLI-MED Corpus: Characteristics, Procedures, and Plans
Table 13: Average annotator confidence in positive judgments per event category, broken down by partial vs. full agreement on videos’ relation to the event.
['Event #', 'Event Name', '% with Partial Agreement', 'Avg. Confidence for Partial', '% with Full Agreement', 'Avg. Confidence for Full', 'Total Avg. Confidence']
The average confidence across all positive videos in every category is 1.95 for videos on which agreement was only partial and 2.77 for videos on which all three annotators agreed they were positive examples; this difference is statistically significant at p < 0.0001 (Student’s unpaired t-test, two-tail). Confidence was also more variable for events with only partial agreement, with a standard deviation of 0.68 across all videos with partial agreement and a standard deviation of 0.48 across videos with full agreement.