Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
Table 3: Ablation study results on HotpotQA dev set.
['[EMPTY]', 'joint EM', 'joint  [ITALIC] F1']
Then we ablate different components in the “Answer and Explain” module, and we measure the model performance in these circumstances in terms of joint EM and F1. (“-mixed attn”) is used, the performance drops marginally, indicating that the mixed attention mechanism helps but in a limited extent. However, replacing attention based sentence summarization with simple averaging (“-attn sum”) deteriorates the performance a lot (the joint EM drops almost 2%), which proves the effectiveness of attention based sentence summarization. For the GNN module, if we do not use GNN based message passing over sentence embeddings, the performance degrades by ∼1.4% for EM and ∼0.9% for F1. If we remove one of the three types of edges, the results show that the first type of edges, i.e. the connection among sentences within the same documents, contributes the most while other two types of edges contribute marginally. But when we remove both of them, the results further drops, indicating the usefulness of cross-document information.