Multi-Mention Learning for Reading Comprehension with Neural Cascades
Table 2: Model ablations on the full Wikipedia development set. For row labeled **, explanation provided in Section §4.3.
['[EMPTY]', 'Wikipedia Dev (EM)']
Our final approach (3-Level Cascade, Multi-loss) achieves the best performance. Training with only a single loss in level 3 (3-Level Cascade, Single Loss) leads to a considerable decrease in performance, signifying the effect of using a multi-loss architecture. It is unclear if combining the two submodels in level 1 into a single feed-forward network that is a function of the question, span and short context (3-Level Cascade, Combined Level 1) is significantly better than our final model. Although we found it could obtain high results, it was less consistent across different runs and gave lower scores on average (49.30) compared to our approach averaged over 4 runs (51.03). Finally, the last three rows show the results of using only smaller components of our model instead of the entire architecture. In particular, our model without the aggregation submodel (Level 1 + Level 2 Only) performs considerably worse, showing the value of aggregating multiple mentions of the same span across the document. As expected, the level 1 only models are the weakest, showing that attending to the document is a powerful method for reading comprehension. Figure 3 (left) shows the behavior of the k-best predictions of these smaller components. While the difference between the level 1 models becomes more enhanced when considering the top-k candidates, the difference between the model without the aggregation submodel (Level 1 + Level 2 Only) and our full model is no longer significant, indicating that the former might not be able to distinguish between the best answer candidates as well as the latter.