Natural Language Comprehension with the EpiReader
Table 2: Model comparison on the CBT and CNN datasets. Results marked with 1 are from Hill et al. (2016), those marked with 2 are from Kadlec et al. (2016), those marked with 3 are from Hermann et al. (2015), and those marked with 4 are from Chen et al. (2016).
['Model', 'CBT-NE valid', 'CBT-NE test', 'CBT-CN valid', 'CBT-CN test']
We measure EpiReader performance at the output of both the Extractor and the Reasoner. Aside from achieving state-of-the-art results at its final output, the EpiReader framework gives a boost to its Extractor component through the joint training process. These are all higher than the analogous scores reported for the AS Reader, which we verified ourselves to within negligible difference. Based on our own work with that model, we found it to overfit the training set rapidly and significantly, achieving training accuracy scores upwards of 98% after only 2 epochs. We suspect that the Reasoner module had a regularizing effect on the Extractor, but leave the verification for future work. An analysis by \newcitekadlec2016 indicates that the trained AS Reader includes the correct answer among its five most probable candidates on approximately 95% of test examples for both datasets. We verified that our Extractor achieved a similar rate, and of course this is vital for performance of the full system, since the Reasoner cannot recover when the correct answer is not among its inputs.