Iterative Alternating Neural Attention for Machine Reading
Table 4: Per-category performance of the Stanford AR and our system. The first three categories require local context matching, the next two global context matching and coreference errors are unanswerable questions [Chen et al.2016].
['Category Exact Match', 'Stanford AR 13', 'Stanford AR (100.0%)', 'Ours 13', 'Ours (100.0%)']
Local context categories generally seem to be easily tackled by the neural models, which perform similarly. It seems that the iterative alternating attention inference is better able to solve more difficult examples such as Ambiguous/Hard. One hypothesis is that, in contrast to Stanford AR, which uses only one fixed-query attention step, our iterative attention may better explore the documents and queries. Finally, Coreference Errors (∼25% of the corpus) includes examples with critical coreference resolution errors which may make the questions “unanswerable”. If this estimate is accurate, our ensemble model (76.1%) may be approaching near-optimal performance on this dataset.