TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation
Table 5: Inter-annotator agreement.
['Annotation', 'Kappa']
To better understand the difficulty of our annotation tasks, we performed inter-annotator agreement experiments for each of our annotations. We randomly sampled twenty-five resources and had annotators label for pedagogical function. Additionally, we sampled twenty-five topics for prerequisite annotations and five topics with reading list lengths of five for survey annotation. We used Fleiss’s Kappa Fleiss et al. designed to measure annotator agreement for more than two annotators. Using the scale as defined in \newciteLandis:1977, pedagogical function annotation exhibits substantial agreement while prerequisite annotation and survey extraction annotation show fair agreement. The Kappa score for pedagogical function is comparable to that of \newciteSheng:17 (0.68) while the prerequisite annotation is slightly lower than the agreement metric used in \newciteGordon:16 (0.36) although they measure agreement through Pearson correlation. We believe that the sparsity of the labels plays a role in these scores.