Teaching Machines to Converse
Table 7.5: Mechanical Turk Task Results. Asking Questions (AQ) outperforms only answering questions without asking (QA).
['[EMPTY]', 'vanilla-MemN2N Task 4: K. Verification', 'vanilla-MemN2N Task 4: K. Verification', 'vanilla-MemN2N Task 8: Triple', 'vanilla-MemN2N Task 8: Triple', 'Cont-MemN2N Task 4: K. Verification', 'Cont-MemN2N Task 4: K. Verification', 'Cont-MemN2N Task 8: Triple', 'Cont-MemN2N Task 8: Triple']
We tune hyperparameters on the validation set and repeat each experiment 10 times and report the best result. We also provide additional experiments. As we collected the data in the same settings as Task 4 and 8 of our simulator, we could also consider supplementing training with simulated data as well, of which we have a larger amount (over 100,000 episodes). Note this is only for training, we will still test on the real (Mechanical Turk collected) data. Although the simulated data has less lexical variety as it is built from templates, the larger size might obtain improve results.