Recipes for building an open-domain chatbot
Table 3: Perplexity of the pre-trained and fine-tuned models on the validation set for BST datasets. Note that perplexity is not directly comparable between the 90M models and the larger models as 90M models use a different dictionary. Fine-tuning gives gains for each skill (task) compared to pre-training on pushshift.io Reddit alone.
['Model', 'Size', 'ConvAI2', 'WoW', 'ED', 'BST', 'Avg.']
We also report perplexity both before and after fine-tuning each of these models on the ConvAI2, Wizard of Wikipedia, Empathetic Dialogues, and Blended Skill Talk datasets. They show that fine-tuning gives relatively large improvements in perplexity on these tasks, which could hence translate into improved ability at these skills when conducting open-domain dialogue.