Teaching Machines to Converse
Table 7.6: Mechanical Turk Task Results, using real data for training and testing.
['[EMPTY]', 'vanilla-MemN2N Task 4: K. Verification', 'vanilla-MemN2N Task 4: K. Verification', 'vanilla-MemN2N Task 8: Triple', 'vanilla-MemN2N Task 8: Triple', 'Cont-MemN2N Task 4: K. Verification', 'Cont-MemN2N Task 4: K. Verification', 'Cont-MemN2N Task 8: Triple', 'Cont-MemN2N Task 8: Triple']
The best results are obtained from the combination of simulator and real data. The best real data only results (selecting over algorithm and training strategy) on both tasks outperform the best results using simulator data, i.e. using Cont-MemN2N with the Train AQ / TestAQ setting) 0.774 and 0.797 is obtained vs. 0.714 and 0.788 for Tasks 4 and 8 respectively. This is despite there being far fewer examples of real data compared to simulator data. Overall we obtain two main conclusions from this additional experiment: (i) real data is indeed measurably superior to simulated data for training our models; (ii) in all cases (across different algorithms, tasks, and data types – be they real data, simulated data or combinations) the bot asking questions (AQ) outperforms it only answering questions and not asking them (QA). The latter reinforces the main result of the paper.