Multitask Models for Supervised Protest Detection in Texts
Table 2: Results for task 1 and 2 model.888All values computed using contest-provided code. Precision and recall values were not provided by the contest organizers in the output of test and China data set evaluations and are therefore unavailable here. Unfortunately, the random seed values for the models trained and evaluated on the test and China data sets were lost and so the (train and dev) and (test and China) results are from two different model runs.
['[EMPTY]', 'Task 1 Precision', 'Task 1 Recall', 'Task 1 F1', 'Task 2 Precision', 'Task 2 Recall', 'Task 2 F1']
The multitask model for tasks 1 and 2 is able to classify documents with 92% accuracy on the dev set with an F1 score of 0.80. This result on the test set is encouraging because it is actually above the corresponding value on the dev set, 0.80, and suggests that the model has not be overfit to the dev set through hyperparameter selection. The model for task 3 is also able to perform document and sentence classification. While test and China set Future work should determine whether this is due to the second model’s ability to overfit to these tasks and data sets or due to the inclusion of task 3 data in the model’s first GRU layer. One approach for exploring this is discussed in the paper’s final section.