Sketch-Fill-A-R: A Persona-Grounded Chit-Chat Generation Framework
Table 3: User study ratings of single-turn responses (score range 1 (lowest) - 5 (highest)). Each experiment showed generated responses from a Sketch-Fill-A-R-variation and KVMemNet on 100 conversations to 5 human raters. Each row shows ratings from a single heads-up experiment. Sketch-Fill with reranking show a small gain over KVMemNet on all qualitative metrics, but the variance in the ratings is high. Sketch-Fill without reranking perform much worse, due to their responses not being fluent, despite achieving low perplexity (see Figure 2).
['Test I', '[BOLD] Baseline KVMemNet', '[BOLD] Consistency 3.60±0.84', '[BOLD] Engagingness 3.81±0.66', '[BOLD] Fluency 4.49±0.45', '[BOLD] Ours Sketch-Fill', '[BOLD] Consistency 2.51±1.16', '[BOLD] Engagingness 2.57±1.10', '[BOLD] Fluency 2.98±1.29']
The study showed small gains on all metrics for all Sketch-Fill-A-R variations, however, the variance of results was high. We believe that this artifact could be caused by a number of factors, including subjective preferences of raters and potential ambiguities in the experiments description. We notice that Sketch-Fill and Sketch-Fill-A reach lower perplexity values than KVMemNet, but comparatively have lower evaluations across the board. Conversely, ranking models like Sketch-Fill-R and Sketch-Fill-A-R have higher scores on all metrics. We observe that the difference is due to the ranker giving more fluent outputs via better selection of persona words to use.