Linguistic Knowledge as Memory for Recurrent Neural Networks
Table 1: Error rates on the 20 bAbi tasks. N2N refers to End-to-end Memory Networks (Sukhbaatar et al., 2015). QRN refers to Query Reduction Networks (Seo et al., 2017b). For MAGE models a number in the parenthesis indicates the dimensions for coreference states in the split output. “shared” in the parenthesis indicates that outputs were shared for all edge types. DAG-RNN refers to the model proposed by Shuai et al. (2016) incorporated with GA. “one-hot” refers to the baseline where coreference features are appended to the input. A task is considered failed if the error rate is above 5%.
['[BOLD] Model', '[BOLD] Task  [BOLD] 1', '[BOLD] Task  [BOLD] 2', '[BOLD] Task  [BOLD] 3', '[BOLD] Task  [BOLD] 4', '[BOLD] Task  [BOLD] 5', '[BOLD] Task  [BOLD] 6', '[BOLD] Task  [BOLD] 7', '[BOLD] Task  [BOLD] 8', '[BOLD] Task  [BOLD] 9', '[BOLD] Task  [BOLD] 10', '[BOLD] Task  [BOLD] 11', '[BOLD] Task  [BOLD] 12', '[BOLD] Task  [BOLD] 13', '[BOLD] Task  [BOLD] 14', '[BOLD] Task  [BOLD] 15', '[BOLD] Task  [BOLD] 16', '[BOLD] Task  [BOLD] 17', '[BOLD] Task  [BOLD] 18', '[BOLD] Task  [BOLD] 19', '[BOLD] Task  [BOLD] 20', '[BOLD] Overall  [BOLD] Avg', '[BOLD] Overall  [BOLD] #failed']
The natural language in the stories was generated using templates, which are easy to learner. Rather, the difficulty of these tasks lies in tracking the state of multiple entities across long distances in the input sequence, which makes them particularly suitable for testing our proposed model with explicit memory. Specifically, we connect consecutive mentions of the same entity with coreference links, which are easily extracted due to the synthetic nature of the language. Our model achieves new state-of-the-art results, outperforming strong baselines such as QRNs. Moreover, we observe that the proposed MAGE architecture can substantially improve the performance for both bi-GRUs and GAs. Adding the same information as one-hot features fails to improve the performance, which indicates that the inductive bias we employ on MAGE is useful. The DAG-RNN baseline from Shuai et al.