Linguistic Knowledge as Memory for Recurrent Neural Networks
Table 2: Performance comparison on bAbi-Mix. We report average error rate over 20 tasks, as well as the error rate on Task 3.
['[BOLD] Model', '[BOLD] Average', '[BOLD] Task 3']
Our motivation in incorporating the DAG structure in text comprehension models is to help the reader model long term dependencies in the input sequences. To test how well it is able to do that, we constructed a new version of the bAbi tasks, called bAbi-Mix, where each story is a mixed version of two independent stories in two different worlds. This was done by first replacing all entity mentions in one of the stories with alternates, so “Daniel” becomes “David”, “milk” becomes “juice” and so on. Then we mixed the sentences in the two stories, in random order, and asked a question about one of the stories. As a result, the stories become longer, and relevant information about entities has to be tracked over longer distances. Answering the questions is still trivial for human readers, but becomes even more challenging for RNN models. Both variants of MAGE substantially outperform QRNs, which are the current state-of-the-art models on the bAbi dataset. As a case study, we also report the performances on Task 3 which requires reasoning over three supporting facts. We observe that MAGE increases the ability to model multiple facts and reduces the error rate from 72.8% to 0.3%.