Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
Table 2: Results of different agents at training epoch = {100, 200, 300}. Each number is averaged over 3 runs, each run tested on 50 dialogues. (Success: success rate, Reward: Average Reward, Turns: Average Turns)
['[BOLD] Agent', '[BOLD] Epoch = 100  [BOLD] Success', '[BOLD] Epoch = 100  [BOLD] Reward', '[BOLD] Epoch = 100  [BOLD] Turns', '[BOLD] Epoch = 200  [BOLD] Success', '[BOLD] Epoch = 200  [BOLD] Reward', '[BOLD] Epoch = 200  [BOLD] Turns', '[BOLD] Epoch = 300  [BOLD] Success', '[BOLD] Epoch = 300  [BOLD] Reward', '[BOLD] Epoch = 300  [BOLD] Turns']
As mentioned in the previous section, a large number of planning steps means leveraging a large amount of simulated experience to train the agents. In contrast, the proposed D3Q framework demonstrates robustness to the number of planning steps Peng et al. From the table, with fewer simulated experiences, the difference between DDQ and D3Q may not be significant, where DDQ agents achieve about 50%-60% success rate and D3Q agents achieve higher than 68% success rate after 100 epochs. However, when the number of planning steps increases, more fake experiences significantly degrade the performance for DDQ agents, where DDQ(10, fixed θG) suffers from bad simulated experiences after 300 epochs and achieves 0% success rate.