iSarcasm: A Dataset of Intended Sarcasm
Table 5: The agreement between intended labels (int.), provided by the authors, and perceived labels, provided by third-party annotators, (perc.) on iSarcasm test set.
['[EMPTY]', 'perc. sarc.', 'perc. non-sarc.']
As we mentioned earlier, we collected three third-party labels for each tweet in the test set of iSarcasm. as a measure, the pairwise inter-annotator agreement (IAA) scores were κ12=0.37, κ13=0.39 and κ23=0.36, which highlights the high subjectivity of the task. We used majority voting to select the final perceived sarcasm label for each tweet. As shown, 30% of the sarcastic tweets were unrecognised by the annotators, while 45% of the tweets perceived as sarcastic were actually not intended to be sarcastic by their authors. This supports our argument that third-party annotation for sarcasm should not be trusted.