Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks
Table 4: Results of ablation study.
['DailyDialog model variant', 'DailyDialog PPL', 'DailyDialog BLEU', 'DailyDialog distinct-1', 'DailyDialog distinct-2', 'DailyDialog Average', 'DailyDialog Greedy', 'DailyDialog Extrema']
First of all, all auxiliary tasks are useful as removing any of them will cause a performance drop. When all auxiliary tasks are removed, the approach degenerates to learning a 2-layer transformer architecture through MLE. Without any optimization on context understanding, the simple structure is worse than ReCoSa. Secondly, on DailyDialog and UDC, order recovery tasks are more crucial than content recovery tasks due to the order insensitive nature of self-attention. Finally, on PERSONA-CHAT, word-level recovery tasks matter more than utterance-level recovery tasks. This might stem from the fact that in PERSONA-CHAT, dialogues highly depend on the profiles used as contexts. In many cases, utterances are just formed by copying a proportion of words from the profiles. Thus, recognizing the semantic connections and the relationship among words in contexts is more critical for the data.