Neural Models for Reasoning over Multiple Mentions using Coreference
Table 1: Accuracy on bAbi-1K, averaged across all 20 tasks. Following previous work we run each task for 10 random seeds, and report the Avg and Max (based on dev set) performance. A task is considered failed if its Max performance is < 0.95.
['[BOLD] Method', '[BOLD] Avg', '[BOLD] Max', '[BOLD] # failed']
Entity-based models. Ji et al. The difference in our work is that we look at the task of reading comprehension, and also work in the more practical setting of system extracted coreferences. Yang et al. Kobayashi et al. Wang et al. also noted the importance of reference resolution for reading comprehension, and we compare our model to their one-hot pointer reader. We also include the results for a single layer version of GA Reader (which we denote simply as Bi-GRU or Bi-C-GRU when using coreference) to enable fair comparison with EntNets. In each case we see clear improvements of using C-GRU layers over GRU layers. Interestingly, EntNets, which have >99% performance when trained with 10K examples only reach 70% performance with 1K training examples. The Bi-C-GRU model significantly improves on this baseline, which shows that, with less data, coreference annotations can provide a useful bias for a memory network on how to read and write memories.