Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
Table 2: Ablation study results on HotpotQA dev set. PR(0,1) stands for giving 0 score to non-gold documents and 1 score to all gold documents when preparing pairwise labels, and PR(0,1,2) stands for giving 2 score to the gold document with answer span.
['[EMPTY]', 'EMS', 'RecallS', 'Accspan', 'joint EM', 'joint  [ITALIC] F1']
We only present the ablation studies on our SAE system using BERT base uncased model. “EMS” and “RecallsS” measure the accuracy and recall of both two gold documents are selected; “Accspan” measures the accuracy of the gold document with answer span being selected. The MHSA mechanism, which allows information from different documents to interact with each other, brings the most gain over the baseline in terms of all measurements. The pairwise learning-to-rank loss function, which allows documents to be compared directly against each other and provides finer-grained information for the model to learn, further improves the performance. Further, scoring documents that contain the answer span to 2 is better than weighting the two gold documents equally. Finally, we can see that better document filter always results in better answer and support sentence prediction performance.