A Multi-Axis Annotation Scheme for Event Temporal Relations
Table 5: An evaluation of MATRES against TB-Dense. Horizontal: MATRES. Vertical: TB-Dense (with interval relations mapped to start-point relations). Please see explanation of these numbers in text.
['[EMPTY]', 'b', 'a', 'e', 'v', 'All']
A few remarks about how to understand it: First, when TB-Dense labels before or after, MATRES also has a high-probability of having the same label (b=455/513=.89, a=309/438=.71); when MATRES labels vague, TB-Dense is also very likely to label vague (v=192/312=.62). This indicates the high agreement level between the two datasets if the interval- or point-based annotation difference is ruled out. Second, many vague relations in TB-Dense are labeled as before, after or equal in MATRES. This is expected because TB-Dense annotates relations between intervals, while MATRES annotates start-points. When durative events are involved, the problem usually becomes more difficult and interval-based annotation is more likely to label vague (see earlier discussions in Sec. extending can be considered durative. Third, equal seems to be the relation that the two dataset mostly disagree on, which is probably due to crowdsourcers’ lack of understanding in time granularity and event coreference. Although equal relations only constitutes a small portion in all relations, it needs further investigation.