Temporal Information Extraction for Question AnsweringUsing Syntactic Dependencies in an LSTM-based Architecture
Table 5: QA results on validation data. There are 79 questions in total. The 4 systems on the top of the table are provided with the toolkit. The systems starting with “human-” are annotated by human experts. TEA-final utilizes both double-check and pruning. TEA-flat uses the flat context. TEA-Dense is trained on TimeBank-Dense.
['[BOLD] system', '[BOLD] coverage', '[BOLD] prec', '[BOLD] rec', '[BOLD] f1']
The results of the four systems above the first horizontal line are provided by the task organizer. Among them, the top two use annotations provided by human experts. As we can see, the precision is very high, both above 0.90. Our models cannot reach that precision. In spite of the lower precision, automated systems can have much higher coverages i.e. answer a lot more questions. As a starting point, we evaluated the validation files in their original form, and the results are shown as “orig. The precision was good, but with very low coverage. This supports our claim that the tlinks provided by the training/validation files are not complete. We also tried using the event and timex tags from the validation data, but performing tlink classification with our system. As shown with “orig. tags TEA tlinks” in the table , now the coverage rises to 64 (or 0.81), and the overall F1 score reaches 0.52. The TEA-initial system uses our own annotators. The performance is similar, with a slight improvement in precision. This result shows our event and timex tags work well, and are not inferior to the ones provided by the training data. We also wanted to check whether a comprehensive annotation of tlinks in the training data can improve model performance on the QA task. Interestingly, the performance is nearly as good as our top model, although TimeBank-Dense only uses five major classes of relations. For one thing, it shows that our system may perform equally after being trained on sparsely labeled data and on densely labeled data, judged from the QA evaluation tool. If this is true, excessively annotated data may not be necessary in some tasks.