Getting Reliable Annotations for Sarcasm in Online Dialogues
Table 3: The 262 items from L&W with initial sarcastic counts in IAC, and initial and final Karger’s labels.
['[BOLD] L&W  [BOLD] SARC COUNT', '[BOLD] INITIAL  [BOLD] NOT-SARC', '[BOLD] INITIAL  [BOLD] SARC', '[BOLD] FINAL  [BOLD] NOT-SARC', '[BOLD] FINAL  [BOLD] SARC']
This provides another view on which items are still ambiguous, i.e. the ones our methods can’t agree on, after collecting 25 annotations. Only approximately 9 out of 600 are still ambiguous (pale blue in Fig. Thus for the subclass of utterances considered AMBIG initially, we achieve reliable labels for these with an additional 15 to 20 annotations. The return for the 15 annotations above 10 is small. Some examples of these types of utterances are provided in Fig. Interestingly, there are also utterances (R4 of Fig. that remain ambiguous even with 30 annotations.