Scale Up Event Extraction Learning via Automatic Training Data Generation
Table 4: System performance when compared against automatic annotations (%).
['[BOLD] Model', '[BOLD] Event Type  [BOLD] P', '[BOLD] Event Type  [BOLD] R', '[BOLD] Event Type  [BOLD] F', '[BOLD] Key Argument Detection  [BOLD] P', '[BOLD] Key Argument Detection  [BOLD] R', '[BOLD] Key Argument Detection  [BOLD] F', '[BOLD] All Argument Detection  [BOLD] P', '[BOLD] All Argument Detection  [BOLD] R', '[BOLD] All Argument Detection  [BOLD] F']
Firstly, we compare the model output against the automatically obtained event annotations. One of the main reasons is the absence of explicit trigger annotations in our dataset, which makes it impossible to include trigger-related features, e.g., trigger-related dependency and position features. Although traditional models can achieve higher precisions, they only identify a limited number of events, resulting in low recalls. Neural-network methods perform much better than feature-based models, especially in recall, since they can make better use of word semantic features. Specifically, BLSTM can capture longer dependencies and richer contextual information, instead of neighbouring word features only. The CRF layer brings an averagely 2% improvement in all metrics, and by adding the ILP-based post inference, our full model, BLSTM-CRF-ILPmulti, achieves the best performance among all models. With the help of constraint C4, dubious key arguments can be correctly inferred through other key arguments from their context. Compared with BLSTM-CRF, BLSTM-CRF-ILP1 produces an F1 gain of 3.2% in event type classification, 1.5% in key argument detection, and 1.2% in all argument detection. BLSTM-CRF-ILPmulti remains the best performing model, achieving 80.7%, 56.4% and 36.3% in F1 scores for event type, key argument detection and all argument detection, respectively.