EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators
Table 4: Annotations Metrics of EDAs - α: Krippendorff’s Alpha coefficient, k: Fleiss’ Kappa score, and SCC: Spearman Correlation between Context-based Models.
['Metrics', '[ITALIC] α', '[ITALIC] k', 'SCC']
We apply it on the five neural annotators at the nominal level of measurement of dialogue act categories. α is computed as follows: α=1−DoDe (4) where Do is the observed disagreement and De is the disagreement that is expected by chance. α=1 means all annotators produce the same label, while α=0 would mean none agreed on any label. Hence, we decided to add one more inter-annotator metric below. The kappa k can be defined as, k=¯P−¯Pe1−¯Pe (5) where the denominator 1−¯Pe elicits the degree of agreement that is attainable above chance, and the numerator ¯P−¯Pe provides the degree of the agreement actually achieved above chance. Hence, k=1 if the raters agree completely, and k=0 when they do not reach any agreement. We got 0.556 and 0.502 for IEOMOCAP and MELD, respectively, with our five neural annotators. This indicates that the annotators are labelling the dialogue acts reliably and consistently. While using the labels, we checked the absolute match between all context-based models and found that their strong correlation indicates their robustness.