Cyberbullying Detection – Technical Report 2/2018 CS AGH
Table 6. The outline of the ambiguous results and the disambiguation scheme for each type of ambiguity.
['[BOLD] Type of Ambiguity', '[BOLD] No. of Cases', '[BOLD] Disambiguation']
There were only four possible situations of ambiguity. Firstly, if one of the annotators selected I don’t know (later IDK or uncertain), or there was no annotation, the remaining two annotators were taken into consideration. Here, if both selected the same answer, we considered the case as solved (however, we still checked those samples in the second stage of annotations for final confirmation). If the results were scrambled, we applied appropriate weighting scheme to propose an initial proposed estimated value (either 1 for harmful or 0 for non-harmful), and asked an additional cyberbullying expert to verify the choice. Weighting was based on each annotator’s average agreement score (the higher the better). Also, when none of the annotators selected IDK but the results were still scrambled, we also applied an appropriate weighting scheme with expert’s verification. Finally, when there were two IDK answers, we considered the remaining non ambiguous answer as a potentially correct, but with a strong voice from the second stage of annotations and the additional expert.