Training Millions of Personalized Dialogue Agents
Table 4: hits@1 results for the best found Transformer architecture on different test sets. FT-PC: Reddit-trained model fine-tuned on the Persona-chat training set. To be comparable to the state of the art on each dataset, results on Persona-chat are computed using 20 candidates, while results on Reddit use 100.
['Training set', 'Validation set Persona-chat', 'Validation set Reddit']
We compare the performance of transformer models trained on Reddit and on Persona-chat on both datasets. This architecture provides a strong improvement over the results of Zhang et al. , jumping from 35.4% hits@1 to 42.1%. Pretraining the model on Reddit and then fine-tuning on Persona-chat pushes this score to 60.7%, largely improving the state of the art. As expected, fine-tuning on Persona-chat reduces the performance on Reddit. However, directly testing on Persona-chat the model trained on Reddit without fine-tuning yields a very low result. This could be a consequence of a discrepancy between the style of personas of the two datasets.