#MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement
Table 4: Inter-annotator agreements for all the annotation tasks.
['[BOLD] Task', '[BOLD] Krippendorff’s  [ITALIC] α']
We evaluated inter-annotator agreements using Krippendorff’s alpha (K-alpha) K-alpha, unlike simple agreement measures, accounts for chance correction and class distributions and can be generalized to multiple annotators. We observe very strong agreements for most of the tasks with a maximum of 0.92 for the relevance task. The least agreement observed was for the hate speech task at 0.78. We chose a straightforward approach of majority decision for label adjudication: if two or more annotators agreed on assigning a particular class label. In cases of discrepancy, the labels were adjudicated manually by the authors.