Structural Embedding of Syntactic Trees for Machine Comprehension
Table 3: Performance comparisons of models with only syntactic information against their counterparts with randomly shuffled node sequences and randomly generated tree nodes using the SQuAD Dev set
['[BOLD] Method', '[BOLD] EM', '[BOLD] F1']
We compared the performance of the models using syntactic information along in their original order (i.e. SECT-Only and SEDT-Only) against their counterparts with the same syntactic tree nodes but with randomly shuffled order (i.e. SECT-Random-Order and SEDT-Random-Order) as well as the baselines with randomly generated tree nodes (i.e. SECT-Random and SEDT-Random). Here we choose the length of window size to be 10. From the table we see that both the ordering and the contents of the syntactic tree are important for the models to work properly: constituency and dependency trees achieved over 20% boost on performance compared to the randomly generated ones and our proposed ordering also out-performed the random ordering. It also worth mentioning that the ordering of dependency trees seems to have less impact on the performance compared to that of the constituency trees. This is because sequences extracted from constituency trees contain hierarchical information, which ordering will affect the output of the model significantly. However, sequences extracted from dependency trees are all children nodes, which are often interchangeable and don’t seem to be affected by ordering much.