Can You Put it All Together:Evaluating Conversational Agents’ Ability to Blend Skills
Table 3: Breakdown of conversations by number of modes, showing that most BST dataset conversations exhibit multiple modes. Workers were asked to choose if each utterance of a conversation demonstrated knowledge, empathy, personal situations, or personal background. Over 70% of the conversations annotated demonstrated at least 3 of the 4 modes.
['Mode Count', 'Conversations', 'Pct (%)']
We also asked crowdsource workers to rate individual utterances as exhibiting one of four possible modes: Knowledge: using factual information (“I’ve heard that in some places, lifeguards also help with other sorts of emergencies, like mountain rescues!”) (dinan2018wizard) Empathy: understanding and acknowledging implied feelings (“I’m sorry to hear that. I wish I could help you figure it out”) (rashkin2019empathy) Personal situations: past circumstances in a person’s life (“I finally got that promotion at work! I have tried so hard for so long to get it!”) (rashkin2019empathy) Personal background: a person’s personality, interests, and attributes (“I am into equestrian sports.”) (zhang2018personalizing) All utterances in over 700 conversations from the validation set of the BST dataset, from both guided and unguided workers, were annotated in this manner for 7,380 annotations collected in total. Workers were able to select as many attributes as they wished for each utterance. To avoid worker-specific bias, each crowdsource worker was limited to performing annotations on 10 conversations, and 123 total workers contributed annotations. Most analysis in this paper refers to three datasets, and the utterance classifier was trained with three dataset labels as classes. However, the ED dataset contains both “Speaker” utterances that describe personal situations, and ”Listener” utterances, where the Listener responds with empathy ( the ED benchmarks trains on both sides but evaluates only on the Listener side). We therefore break down annotations into four types, with two types covering responses about “personal topics”: personal background (which is the focus of ConvAI2) and personal situations (talked about in ED). Over 70% of conversations annotated contained at least 3 of 4 modes. Overall, workers’ annotation counts are 43.7% for personal background, 20.5% for knowledge, 20.3% for empathy, and 15.4% for personal situations. This supports the finding from our utterance classifier that the vast majority of conversations feature more than one mode, where utterance modes are defined as the predicted dataset provenance per utterance. In order to avoid excessive annotator bias and keep annotations discriminative, we limit the maximum number of annotations per worker and check that annotators did not select all modes for each utterance.