Pattern Learning for Detecting Defect Reports and Improvement Requests in App Reviews
Table 2: Inter-annotator agreement per task
['Question', 'Agreement(%)']
We collected annotations for both feedback types through CrowdFlower (recently renamed Figure Eight), an online data enrichment platform. The instructed task is to label every individual review for both defect reports and improvement requests. Every review was annotated by at least 3 annotators, and in some cases even 5 or 7 (when it is recorded a low accuracy of the test questions that inspect the quality of the annotator). In order to estimate the reliability of the aggregated annotations, we computed the inter-annotator agreement (IAA) for each question, using Fleiss’ Kappa measure. We use the majority voting for building our gold standard.