Multitask Models for Supervised Protest Detection in Texts
Table 4: Role-wise performance on task 3 dev set
['[EMPTY]', 'Precision', 'Recall', 'F1']
These metrics are evaluated on the out-of-sample dev set. The model performs better on common role labels than less common labels; it achieves F1 scores greater than 0.5 on triggers, participants, and places. The model fails to label any locations correctly. This is probably due to the model’s failure to recognize the prepositions preceding locations as the B tokens in the location phrase. For example, “in front of the state secretariat” should be labeled “B-loc, I-loc, I-loc, I-loc, I-loc, I-loc.” Instead, the model predicts “O, I-loc, I-loc, I-loc, I-loc, I-loc.” Another example from the dev set reads “near a mosque” and should be labeled “B-loc, I-loc, I-loc.” The model instead predicts “B-fname, I-loc, I-loc,” where “fname” represents the role “facility name.”