Personalizing Dialogue Agents: I have a dog, do you have pets too?
Table 6: Evaluation of dialog utterance prediction with ranking models using hits@1 in four settings: conditioned on the speakers persona (”self persona”), the dialogue partner’s persona (”their persona”), both or none. The personas are either the original source given to Turkers to condition the dialogue, or the rewritten personas that do not have word overlap, explaining the poor performance of IR in that case.
['Method', 'No Persona Orig', 'No Persona Rewrite', 'Self Persona Orig', 'Self Persona Rewrite', 'Their Persona Orig', 'Their Persona Rewrite', 'Both Personas Orig', 'Both Personas Rewrite']
Revised Personas. Revised personas are much harder to use. We do however still see some gain for the Profile Memory networks compared to none (0.354 vs. 0.318 hits@1). Training on revised personas helps, both for test examples that are in original form or revised form, likely due to the model be forced to learn more than simple word overlap, forcing the model to generalize more (i.e., learn semantic similarity of differing phrases). Their Persona. Using “Their persona” has less impact on this dataset. We believe this is because most speakers tend to focus on themselves when it comes to their interests. It would be interesting how often this is the case in other datasets. Certainly this is skewed by the particular instructions one could give to the crowdworkers. For example if we gave the instructions “try not to talk about yourself, but about the other’s interests’ likely these metrics would change. In particular we give results for conditioning on the other person’s persona (”Their”) instead of one’s own (”Self”) or the combination (”Both”). In particular, we see that, knowing the other’s persona does not help precision on this data with these models. Revised personas give improved results, perhaps because the models are forced to learn more than trivial word overlap (i.e. semantic similarity of differing phrases).