Getting Reliable Annotations for Sarcasm in Online Dialogues
Table 2: The number of items that switch labels from sarcastic (+) to not sarcastic (-) after 23 annotations given how many annotations you start with. The mean indicates the average ratio of sarcastic to non sarcastic votes based on Karger’s method for that label. The max and min values represent largest (or smallest) value in the set (i.e., the most confident in the wrong label).
['Start 3', '+− 18', 'Mean 0.53', 'Max 0.60', '−+ 19', 'Mean 0.48', 'Min 0.44']
The table also shows the mean ratio for the items that switched label and the maximum (for +−) and minimum (for −+) values. On average, only the items that start out highly ambiguous with only a few labels change labels when over 20 annotations are provided. It also shows that as the number of starting annotations increases the mean and max values stabilize around 7 annotations, with a few minor fluctuations. Our conclusion from this data is that in most cases we only need to acquire 3 annotations and can increase the number up to 7 when the initial judgments are highly ambiguous.