Learning End-to-End Goal-Oriented Dialog with Multiple Answers
Table 3: Ablation study of our proposed model on permuted-bAbI dialog task. Results (accuracy %) are given in the standard setup, without match-type features.
['[BOLD] Model', '[BOLD] Per-turn', '[BOLD] Per-dialog']
Here, we study the different parts of our model for better understanding of how the different parts influence the overall model performance. We show results for Mask-memN2N in various settings - a) without entropy, b) without pre-training mask c) reinforcement learning phase only. When we do only the RL phase, it might be very tough for the system to learn everything by trial and error, especially because the action space is so large. Preceding it with the SL phase and L2 mask pre-training would have put the system and its parameters at a good spot from which the RL phase can improve performance. Note that it would not be valid to check performance of the SL phase in the test set as the SL phase requires the actual answers for it to create the mask.