Retrieve and Refine: Improved Sequence Generation Models For Dialogue
Table 4: Human Evaluation scores for the models,scoring fluency, engagingness, consistency and persona detection, with standard deviation in parentheses. We consider engagingness to be the most important metric.
['Comparison (A vs. B)', 'Win Rate', 'A Wins', 'B Wins', 'Tie', 'p-value']
They show engagingness scores superior to Seq2Seq for all RetNRef variants, and with RetNRef++ slightly outperforming the retriever which it conditions on. Importantly however, it maintains this performance whilst still being able to generate text which a retrieval model cannot. It also performs well in the other metrics, although like the Memory Network model, it is weaker at using the persona than Seq2Seq. Seq2Seq is inferior to the Memory Network Retriever in terms of engagement, in line with previous results. We also tried overtraining the Seq2Seq for 100 epochs instead of early stopping by validation on perplexity as it may overfit training sentences and act more as a retriever, but this did not help. To compute a statistically more meaningful test, and to evaluate models more clearly against each other, we also tested models by pairing them against each other (different to the protocol previously used for this dataset). We took human-human paired dialogues from the test set, up to a randomly chosen turn, and then computed the response of two rival models. The resulting dialogue history and the two choices are given to a third annotator to evaluate and asked which model is best; they can also choose “unsure”. We collected over 1000 such examples, with each conversation being evaluated twice.