Public Self-consciousness for Endowing Dialogue Agents with Consistent Persona
Figure 3: Performance variation of the self-conscious agents according to β and different updates for world prior pt(i).
['[BOLD] PersonaChat Model', 'LostInConv H@1 ↑', 'LostInConv F1 ↑', 'LostInConv PPL ↓', 'LostInConv C ↑', 'Transfer-T H@1 ↑', 'Transfer-T F1 ↑', 'Transfer-T PPL ↓', 'Transfer-T C ↑']
Insensitivity to Contradictory Utterances. From quantitative evaluation, we reveal existing generative models for dialogues are highly insensitive to contradictory utterances. We report four ranking metrics following Welleck et al. Each is the proportion of the GT, entailing, neutral and contradictory utterances in the top-1 candidate chosen by the model, respectively. The models rank the candidates by perplexity scores. Contradict@1 is even higher than the sum of Hits@1 and Entail@1 (32.9, 37.5). Quantitative Results. In Dialogue NLI, our self-conscious S1 significantly reduces Contradict@1 score and increases the Entail@1 along with Hits@1 of S0. Since each entailing candidate shares the same annotated triple as the GT utterance, Entail@1 is a lenient version of Hits@1 Our Distractor Memory significantly improves the vanilla S1 models across all metrics. World Prior Update. Our experiments confirm that updating the world prior with Lt1 makes no difference to the performance compared with using a uniform distribution; this was first reported in Cohn-Gordon et al. The pragmatic listener Lt1∝St0(ut|i,h,u