Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Table 8: Reranking performance with different ablations. yhard and ysoft refer to the two labels used to train the reranker.
['Model', 'TriviaQA-Wikipedia EM', 'TriviaQA-Wikipedia F1', 'SQuAD-document EM', 'SQuAD-document F1']
As we can see, removing the non-maximum suppression (NMS) algorithm has a negative impact on the performance, suggesting it is necessary to prune highly-overlapped candidate answers before reranking. Ablating the hard label leads to a drop of 0.81 and 0.64 F1 scores on two datasets respectively, while the F1 drops by 0.39 and 0.76 points after removing the soft label. This implies that the hard label has a larger impact than the soft label on the TriviaQA dataset, but vice versa on SQuAD.