Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
Table 9: QA+Recommendation task test results (h@10 metric). The last row shows MemN2N without access to a long-term memory (KB).
['[EMPTY]', 'Whole', 'Response 1', 'Response 2', 'Response 3']
S4SS0SSS0Px3 Using Dialog History In both QA+Recommendations (Task 3) and Reddit (Task 4) Memory Networks outperform Supervised Embeddings due to their better use of context. Note that these improvements come from the short term memory (dialog history), not from the use of the KB, as we show Memory Networks results without access to the KB and they perform similarly. We believe the QA performance in these cases is not hindered by the lack of a KB because we ask questions based on fewer relations than in Task 1 and it is easier to store the knowledge directly in the word embeddings. LSTMs perform poorly: the posts in Reddit are quite long and the memory of the LSTM is relatively short, as pointed out by Sordoni et al. In that work they employed a linear reranker that used LSTM prediction as features to better effect. Testing more powerful recurrent networks such as LSTMs with attention on these benchmarks remains as future work (although the latter is related to Memory Networks, which we do report).