Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Table 4: Ablation study on the development set.
['Model', 'LA. F1', 'SA. F1']
To evaluate the effectiveness of our proposed model, we conduct an ablation study on the development dataset on the BERT-base setting. First, we discuss the effect of the joint training strategy. We can see that the removal of either sub-task goals will bring decreases on both tasks. It suggests that the two-grained answers can promote each other with our multi-granularity representation. Then we remove the whole graph module, which means the inference process depends on the score of short answer spans because long answer candidates cannot be scored. We can see the decrease of both long and short answer performance by 5.0% and 0.9%, respectively, indicating the effectiveness of our proposed graph representations. Finally, we investigate the effect of components in our graph encoder. When removing the graph integration layer, the performance of long answer and short answer both become worse by 0.6% and 0.8%. At last, we remove the self-attention layer in the graph encoder, the performance of long answer and short answer both become worse by 0.5% and 0.7%. The ablation study shows the importance of each component in our method.