Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
Table 5: Comparison of 6 model variants with only supervised learning training.
['[BOLD] Deal', 'PPL', 'Reward', 'Agree%', 'Diversity']
Llite outperforms Lfull as a pre-train objective. We attribute this to the exposure bias in the latent space, i.e. the decoder is not trained to consider the discrepancy between the posterior network and actual dialog policy network. Meanwhile, the full models tend to enjoy higher diversity at pre-training, which agrees with the diversity-promoting effect observed in prior research Zhao et al. This is fundamentally different from diversity in pre-training, since diversity in LaRL is optimized to improve task reward, rather than to better model the original data distribution.