Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Table 3: Test accuracy (%) on our 20 Tasks for various methods (1000 training examples each). Our proposed extensions to MemNNs are in columns 5-9: with adaptive memory (AM), N-grams (NG), nonlinear matching function (NL), and combinations thereof. Bold numbers indicate tasks where our extensions achieve ≥95% accuracy but the original MemNN model of Weston et al. (2014) did not. The last two columns (10-11) give extra analysis of the MemNN\em AM + NG%  + NL method. Column 10 gives the amount of training data for each task needed to obtain ≥95% accuracy, or FAIL if this is not achievable with 1000 training examples. The final column gives the accuracy when training on all data at once, rather than separately.
['[EMPTY]', 'Weakly Supervised', 'Weakly Supervised', 'Uses External Resources', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)', 'Strong Supervision (using supporting facts)']
Learning rates and other hyperparameters for all methods are chosen using the training set. We give results for each of the 20 tasks separately, as well as mean performance and number of failed tasks in the final two rows. The adaptive approach gives a straight-forward improvement in tasks 3 and 16 because they both require more than two supporting facts, and also gives (small) improvements in 8 and 19 because they require multi-word outputs (but still remain difficult). We hence use the AM model in combination with all our other extensions in the subsequent experiments.