Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation
Table 2: RL-Q-IG-NA performs best in PMR and RL-Q-IG perform best in perplexity
['Model', 'PMR', 'Perplexity']
By decoupling the policy learning from the language generation and alternatively optimizing the dialog policy and language model, our RL-Q-IG avoids language deviation while still achieving an optimal dialog policy for the image retrieval task. To further evaluate the contribution from the RL and alternative training curriculum, we conduct two ablation studies. RL-Q-IG-NA is fine-tuned with a policy learning objective that excludes alternatively applying the language model loss. While RL-Q-IG-NA only achieves an incremental improvement over the full framework RL-Q-IG in terms of the PMR rate with less than 0.1%, it suffers from a dramatic increase of perplexity from 61.42 to 363.88, thus suggesting that alternatively applying the supervised learning objective can prevent the language model from deviating from the human language distribution. We additionally apply policy learning on the question decoder of SL-Q-IG and follow the RL fine-tuning process in Das et al. While applying word-level RL enables RL-Q to achieve a moderate improvement over SL-Q in terms of PMR, we did not observe, the same degree of advantage in RL-Q-IG-W over SL-Q-IG. Additionally, RL-Q-IG-W is affected by a marginal increase in perplexity in comparison to the SL pre-trained agent, which approves the drawbacks of applying RL on a large action space in language generation.