Exploiting Partially Annotated Data for Temporal Relation Extraction
Table 2: Performance of various usages of the partially annotated data in training. F: Fully annotated data. P: Partially annotated data. PFull: P with missing annotations filled by vague. PEmpty: P with all annotations removed. Bootstrap: referring to specific implementations of Line 1 in Algorithm 1, i.e., local or global. Same/nearby sentence: edges whose nodes appear in the same/nearby sentences in text. Overall: all edges. Awareness: the temporal awareness metric used in the TempEval3 workshop, measuring how useful the predicted graphs are UzZaman et al. (2013). System 7 can also be considered as a reproduction of Ning et al. (2017) (see the discussion in Sec. 5 for details).
['No.', 'Training Data', 'Training Bootstrap', '[ITALIC] Same Sentence P', '[ITALIC] Same Sentence R', '[ITALIC] Same Sentence F', '[ITALIC] Nearby Sentence P', '[ITALIC] Nearby Sentence R', '[ITALIC] Nearby Sentence F', '[BOLD] Overall P', '[BOLD] Overall R', '[BOLD] Overall F', '[BOLD] Awareness P', '[BOLD] Awareness R', '[BOLD] Awareness F']
We also adopted the original split of TB-Dense (22 documents for training, 5 documents for development, and 9 documents for test). Learning parameters were tuned to maximize their corresponding F-metric on the development set. Using the selected parameters, systems were retrained with development set incorporated and evaluated against the test split of TB-Dense (about 1.4K relations: 0.6K vague, 0.4K before, 0.3K after, and 0.1K for the rest). PFull is a variant of P by filling its missing edges by vague. Since it labels too many vague TempRels, System 2 suffered from a low recall. In contrast, P does not contain any vague training examples, so System 3 would only predict specific TempRels, leading to a low precision. Given the obvious difference in F and PFull, System 4 expectedly performed worse than System 1. However, when we see that System 5 was still worse than System 1, it is surprising because the annotated edges in P are correct and should have helped. This unexpected observation suggests that simply adding the annotated edges from P into F is not a proper approach to learn from both. While incorporating transitivity constraints in inference is widely used, Ning et al. One of the algorithms proposed in Ning et al. Despite the technical similarity, this work is motivated differently and is set to achieve a different goal: Ning et al.