End-To-End Memory Networks
Table 1: Test error rates (%) on the 20 QA tasks for models using 1k training examples (mean test errors for 10k training examples are shown at the bottom). Key: BoW = bag-of-words representation; PE = position encoding representation; LS = linear start training; RN = random injection of time index noise; LW = RNN-style layer-wise weight tying (if not stated, adjacent weight tying is used); joint = joint training on all tasks (as opposed to per-task training).
['[EMPTY]', 'Baseline Strongly', 'Baseline', 'Baseline', 'MemN2N', 'MemN2N', 'MemN2N', 'MemN2N PE', 'MemN2N 1 hop', 'MemN2N 2 hops', 'MemN2N 3 hops', 'MemN2N PE', 'MemN2N PE LS']
They show a number of interesting points: [leftmargin=0cm,itemindent=.5cm,labelwidth =labelsep=0cm,align=left] The best MemN2N models are reasonably close to the supervised models (e.g. 1k: 6.7% for MemNN vs 12.6% for MemN2N with position encoding + linear start + random noise, jointly trained and 10k: 3.2% for MemNN vs 4.2% for MemN2N with position encoding + linear start + All variants of our proposed model comfortably beat the weakly supervised baseline methods. The position encoding (PE) representation improves over bag-of-words (BoW), as demonstrated by clear improvements on tasks 4, 5, 15 and 18, where word ordering is particularly important. The linear start (LS) to training seems to help avoid local minima. Joint training on all tasks helps. Importantly, more computational hops give improved performance. We give examples of the hops performed (via the values of eq. over some illustrative examples in Fig.