Response Generation by Context-aware Prototype Editing
Table 2: Automatic evaluation results. Numbers in bold mean that improvement from the model on that metric is statistically significant over the baseline methods (t-test, p-value <0.01). κ denotes Fleiss Kappa [Fleiss1971], which reflects the agreement among human annotators.
['[EMPTY]', '[BOLD] Relevance Average', '[BOLD] Relevance Extrema', '[BOLD] Relevance Greedy', '[BOLD] Diversity Distinct-1', '[BOLD] Diversity Distinct-2', '[BOLD] Originality Not appear', '[BOLD] Fluency +2', '[BOLD] Fluency +1', '[BOLD] Fluency 0', '[BOLD] Fluency Avg', '[BOLD] Fluency  [ITALIC] κ']
Our methods are better than retrieval-based methods on embedding based metrics, that means revised responses are more relevant to ground-truth in the semantic space. Our model just slightly revises prototype response, so improvements on automatic metrics are not that large but significant on statistical tests (t-test, p-value <0.01). Two factors are known to cause Edit-1-Rerank worse than Retrieval-Rerank. 1) Rerank algorithm is biased to long responses, that poses a challenge for the editing model. 2) Despite of better prototype responses, a context of top-1 response is always greatly different from current context, leading to a large insertion word set and a large deletion set, that also obstructs the revision process. In terms of diversity, our methods drop on distinct-1 and distinct-2 in a comparison with retrieval-based methods, because the editing model often deletes special words pursuing for better relevance. Retrieval-Rerank is better than retrieval-default, indicating that it is necessary to rerank responses by measuring context-response similarity with a matching model.