Can You Put it All Together:Evaluating Conversational Agents’ Ability to Blend Skills
Table 6: Test results on BlendedSkillTalk. BST, zero-shot: the models are tested directly on the test set of BST without having been fine-tuned on the BST train set. +BST, FT: models are fine-tuned on the BST train set, then tested on the BST test set. Multi-Task Single-Skills + BlendedSkillTalk performs best. The Multi-Task Two-Stage model outperforms two of the single-skill models, but the latter work well when combined with BlendedSkillTalk fine-tuning. We hypothesize that ConvAI2 alone performs well because it has been trained to use persona contexts, that are used throughout the BST dialogues.
['Model', 'BST, zero-shot', '+BST, FT']
We show two types of results on the BlendedSkillTalk benchmark (BST). Single-skill models are tested directly on BST without any additional training in a zero-shot setting, or fine-tuned on the BST training set then tested on the BST test-set. The Multi-Task Single-Skills model outperforms all single-skill model baselines, whether used in a zero-shot or fine-tuned fashion, despite being the same size. The MT Two-Stage and Random-Skill models outperform two of the three single-skill models. We hypothesize that the ConvAI2 model is doing better because it has already learned to use personas. All single-skill models show improved performance once fine-tuned on the BST train set. However, performance in the zero-shot setting is already good, which is promising in terms of generalization to unseen data.