Structured Attention Networks
Table 4: Answer accuracy (Ans %) and supporting fact selection accuracy (Fact %) of the three QA models on the 1K bAbI dataset. K indicates the number of hops/inference steps used for each task. Task 7 and 8 both contain variable number of facts and hence they are excluded from the fact accuracy measurement. Supporting fact selection accuracy is calculated by taking the average of 10 best runs (out of 20) for each task.
['Task', '[ITALIC] K', 'MemN2N Ans %', 'MemN2N Fact %', 'Binary CRF Ans %', 'Binary CRF Fact %', 'Unary CRF Ans %', 'Unary CRF Fact %']
For correct answer seletion (Ans %), we find that MemN2N and the Binary CRF model perform similarly while the Unary CRF model does worse, indicating the importance of including pairwise potentials. Since ground truth supporting facts are provided for each query, we can check the sequence accuracy of supporting facts for each model (i.e. the rate of selecting the exact correct sequence of facts) by taking the highest probability sequence ^z = argmaxp(z1,…,zK|x,q) from the model and checking against the ground truth. Overall the Binary CRF is able to recover supporting facts better than MemN2N. This improvement is significant and can be up to two-fold as seen for task 2, 11, 13 & 17. However we observed that on many tasks it is sufficient to select only the last (or first) fact correctly to predict the answer, and thus higher sequence selection accuracy does not necessarily imply better answer accuracy (and vice versa). For example, all three models get 100% answer accuracy on task 15 but have different supporting fact accuracies.