Personalizing Dialogue Agents: I have a dog, do you have pets too?
Table 13: Profile Prediction. Error rates are given for predicting either the persona of speaker 0 (Profile 0) or of speaker 1 (Profile 1) given the dialogue utterances of speaker 0 (PERSON 0) or speaker 1 (PERSON 1). This is shown for dialogues between humans (PERSON 0) and either the KV Profile Memory model (“KV Profile”) which conditions on its own profile, or the KV Memory model (“KV w/o Profile”) which does not.
['[BOLD] Speaker', '[BOLD] Profile', '[BOLD] Profile Level KV Profile', '[BOLD] Profile Level KV w/o Profile', '[BOLD] Sentence Level KV Profile', '[BOLD] Sentence Level KV w/o Profile']
In this section we conduct a preliminary study of the ability to predict the persona of a speaker given a set of dialogue utterances. We tested the ability to predict the profile information of the two speakers from the dialogue utterances of each speaker, considering all four combinations. We employ the same IR baseline model used in the main paper to predict profiles: it ranks profile candidates, either at the entire profile level ( considering all the sentences that make up the profile as a bag) or at the sentence level (each sentence individually). We consider 100 negative profile candidates for each positive profile, and compute the error rate of predicting the true profile averaged over all dialogues and candidates. The results indicate the following: It is possible to predict the humans profile from their dialogue utterances (PERSON 0, Profile 0) with high accuracy at both the profile and sentence level, independent of the model they speaking to. Similarly the model’s profile can be predicted with high accuracy from its utterances (PERSON 1, Profile 1) when it is conditioned on the profile, otherwise this is chance level (w/o Profile). It is possible to predict the model’s profile from the human’s dialogue, but with a lower accuracy (PERSON 0, Profile 1) as long as the model is conditioned on its own profile. This indicates the human responds to the model’s utterances and pays attention to the model’s interests. Similarly, the human’s profile can be predicted from the model’s dialogue, but with lower accuracy. Interestingly, the model without profile conditioning is better at this, perhaps because it does not concentrate on talking about itself, and pays more attention to responding to the human’s interests. There appears to be a tradeoff that needs to be explored and understood here.