Structural Embedding of Syntactic Trees for Machine Comprehension
Table 1: Performance comparison on the development set. Each setting contains five runs trained consecutively. Standard deviations across five runs are shown in the parenthesis for single models. Dots indicate the level of significance.
['[BOLD] Method', '[BOLD] Single  [BOLD] EM', '[BOLD] Single  [BOLD] EM', '[BOLD] Single  [BOLD] F1', '[BOLD] Single  [BOLD] F1', '[BOLD] Ensemble  [BOLD] EM', '[BOLD] Ensemble  [BOLD] F1']
We first compared the performance of single models between the baseline approach BiDAF and the proposed SEST approaches, including SEPOS, SECT-LSTM, SECT-CNN, SEDT-LSTM, and SEDT-CNN, on the development dataset of SQuAD. For each model, we conducted 5 different single experiments and evaluated them using two metrics: “Exact match” (EM), which calculates the ratio of questions that are answered correctly by strict string comparison, and the F1 score, which calculates the harmonic mean of the precision and recall between predicted answers and ground true answers at the character level. SECT-LSTM is the second best method, which confirms the predictive powers of different types of syntactic information. We could see that SEDT-LSTM model outperforms the baseline method and other proposed methods in terms of both EM and F1. Another observation is that our propose models achieve higher relative improvements in EM scores than F1 scores over the baseline methods, providing the evidence that syntactic information can accurately locate the boundaries of the answer.