The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
Table 3: Transfer performance of various multi-task models (validation perplexity).
['Model', 'angle=45,lap=0pt-(1em)ConvAI2', 'angle=45,lap=0pt-(1em)Wiz. of Wikipedia', 'angle=45,lap=0pt-(1em)EmpatheticDialog']
We first perform a preliminary study on a subset of the tasks: Reddit, ConvAI2, Wizard of Wikipedia and Empathetic Dialogues, and report the transfer ability of training on some of them, and testing on all of them (using the validation set), reporting perplexity. They show that training on pushshift.io Reddit alone, a huge dataset, is effective at transfer to other tasks, but never as effective as fine-tuning on the task itself. Moreover, fine-tuning on most of the smaller tasks actually provides improvements over pushshift.io Reddit training alone at transfer, likely because the three tasks selected are more similar to each other than to pushshift.io Reddit. Finally, training on all four tasks is the most effective strategy averaged over all tasks compared to any other single model, although this does not beat switching between different fine-tuned models on a per-task basis. The performance is impressive, with some tasks yielding lower perplexity than BERT pre-training + single task fine-tuning. However, it still lags significantly behind fine-tuning applied after pushshift.io Reddit pre-training.