USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Table 2: Average scores for the six different responses, on the six quality: Understandable, Natural, Maintains Context, Interesting, Uses Knowledge and Overall Quality.
['[BOLD] System', '[BOLD] Und (0-1)', '[BOLD] Nat (1-3)', '[BOLD] MCtx (1-3)', '[BOLD] Int (1-3)', '[BOLD] UK (0-1)', '[BOLD] OQ (1-5)']
Across both datasets and all qualities, the new human generated response strongly outperforms all other response types, even the original ground truth. This may be because the new human generated response was written with this quality annotation in mind, and as such is optimized for turn-level evaluation. On the other hand, the workers who produced the original ground-truth response, were more concerned with the quality of the overall dialog than with the quality of each individual response.