Getting Reliable Annotations for Sarcasm in Online Dialogues
Table 1: Summary of the relationship between L&W labels and Karger’s labels on the L&W13 dataset.
['[BOLD] L&W13', '[BOLD] Karger  [BOLD] AMBIG', '[BOLD] Karger  [BOLD] NOT-SARC', '[BOLD] Karger  [BOLD] SARC']
One of our primary motivations is to explore question O1, i.e. what different reliability models indicate about L&W’s experimental method and results. In L&W none of the examples were originally classified as AMBIG. Note the large number of examples (1125) categorized as SARC by L&W but categorized as NOT-SARC by Karger. This is because Karger treats the SARC label and the NON-SARC label the same, and in many cases the majority of annotators may have used the NON-SARC label. The assumption behind the L&W threshhold (two annotators said it was SARC) Fig. Remember that L&W threshhold was two annotators. The upper part Fig.