Cyberbullying Detection – Technical Report 2/2018 CS AGH
Table 5. The outline of the dataset after the first stage of annotation.
['[BOLD] Type', '[BOLD] Number', '[BOLD] Percentagen']
I Stage of Annotation After the analysis of annotator performance, we looked at the data from the point of view of the annotated samples. Therefore, we focused on agreements calculated per sample, and not per annotator. Most of the samples (almost 90%) were annotated unequivocally, which was a positive result suggesting good quality of the dataset and supporting the good performance of annotators. Grand majority (86%) of annotations was considered as not harmful. Only about 2% was considered harmful without a doubt by all annotators in the first stage of annotations. There was also a small number (28) of samples for which none of the annotators had any idea how to classify them (uncertain / I don’t know), and also a small number of samples (17) which were not annotated, probably by accident. The 28 samples which all three annotators were not able to annotate as well as the missing annotations were forwarded to a cyberbullying expert for final decision. The remaining 1452 samples (11.37%) annotated equivocally with some level of disagreement were used in the second stage of annotation.