Multitask Models for Supervised Protest Detection in Texts
Table 3: Results for task 3 model.999All values computed using contest-provided code. Due to time constraints imposed by the contest structure, performance was not evaluated for tasks 1 and 2 on the test and China data sets. Unfortunately, the random seed values for the models trained and evaluated on the test and China data sets were lost and so the (train and dev) and (test and China) results are from two different model runs.
['[EMPTY]', 'Task 1 Precision', 'Task 1 Recall', 'Task 1 F1', 'Task 2 Precision', 'Task 2 Recall', 'Task 2 F1', 'Task 3 Precision', 'Task 3 Recall', 'Task 3 F1']
However, I focus here on the results for semantic role labeling as this model was only evaluated on the test and China data sets for that particular task. The precision, recall, and F1 scores shown here are multiclass weighted averages computed with a Lab-provided script. The unweighted average accuracy of this model on the dev set is very high, 94%, due largely to class imbalance. The model correctly predicts that most words in each sentence are not one of the selected roles. However, the model appears to generalize poorly: the F1 score for the train data set is 0.82 but drops to 0.50, 0.52, and 0.39 for the dev, test, and China data sets, respectively. This is indicative of a model that is overfit to the training data.