Emotion Detection on TV Show Transcripts withSequence-based Convolutional Neural Networks
Table 3: Inter-annotator agreement (in %).
['[BOLD] # of annotators', '[BOLD] 2', '[BOLD] 3', '[BOLD] 4']
First, Cohen’s kappa is used to measure the agreement between two annotators whereas Fleiss’ kappa is used for three and four annotators. Second, the partial agreement (an agreement between any pair of annotators) is measured to illustrate the improvement from a fewer to a greater number of crowd workers. Among all kinds of annotator groups, the kappa scores around 14% are achieved. Such low scores are rather expected because emotion detection is highly subjective so that annotators often judge different emotions that are all acceptable for the same utterance. This may also be attributed to the limitation of our dataset; a higher kappa score could be achieved if the annotators were provided with a multimodal dataset (e.g., text, speech, image).