Facebook Reaction-Based Emotion Classifier as Cue for Sarcasm Detection
Table 3. Details of the testing sets to be used for evaluation.
['[BOLD] Language', '[BOLD] Set', '[BOLD] # of Texts', '[BOLD] # of Annotators', '[BOLD] Fleiss’ Kappa']
As mentioned previously, one of the main challenges in the sarcasm detection task is the difficulty even for humans to identify sarcastic expressions. It is required to have annotated testing sets of good quality and consistency in order to perform a proper evaluation. We therefore generated several testing sets, 4 for English evaluation and 3 for Chinese. The inter-annotator consistency of out datasets is measured by Fleiss’ Kappa. A good inter-annotator score guarantees the quality of the testing sets, while maintaining this quality across sets is an indicator of consistency. The annotators for the English and Chinese Test 1, 2 ,3 sets are university students between 22 and 29 years old, native speakers of the language being evaluated. The subjects are familiar to the social media platforms and the sarcastic posting behavior in it. According to the suggested interpretation all sets achieve at least Substantial agreement(0.61-0.80).