Cyberbullying Detection – Technical Report 2/2018 CS AGH
Table 1. The number of correctly annotated samples (out of 30) for the people that performed the annotation.
['[BOLD] ID', '[BOLD] Correct Labels', '[BOLD] Percentage']
Sixteen (16) initial candidates responded to our call. The candidates were given an initial test to eliminate possible outliers and low performance annotators. In the initial test the candidates were given 30 random samples to annotate with already prepared gold standard answers. The top eight candidates were retained, while the low performance half was rejected.