[
Table 9: Evokes versus associated: Agreement at two intensity levels of emotion (emotive and non-emotive). Percentage of terms in the pilot set for which the majority class size was 5.
['[BOLD] Emotion', '[BOLD] Majority class size five evokes', '[BOLD] Majority class size five associated']
As alluded to earlier, we performed two separate sets of annotations on the pilot set: one where we asked if a word evokes a certain emotion, and another where we asked if a word is associated with a certain emotion. the the percentage of times all five annotators agreed with each other on the classification of a term as emotive, for the two scenarios. Observe that the agreement numbers are markedly higher with associated than with evokes for anger, anticipation, joy, and surprise. In case of fear and sadness, the agreement is only slightly better with evokes, whereas for trust and disgust the agreement is slightly better with associated. Overall, associated leads to an increase in agreement by more than 5 percentage points over evokes. Therefore all subsequent annotations were performed with associated only.