A system for the 2019 Sentiment, Emotion and Cognitive State Task of DARPA’s LORELEI project
TABLE V: Model performance on English train data estimated using 10-fold CV
['Model', 'Prec', 'Recall', 'F1']
While all the models outperformed the baselines, not all of them did so with a significant margin due to the robustness of the baselines selected. The ones found to be significantly better than the baselines were models IIb (Domain-specific) and IIc (Twitter-only) (permutation test, n=105 both p<0.05). The difference in precision between model IIb and IIc points out to the former making the wrong predictions for news articles. These errors are most likely in selecting the wrong supporting segment. Moreover, even though models IIa-c only produce negative labels, they still achieve improved performance over the state-of-the-art systems, highlighting the highly skewed nature of the training dataset.