USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Table 1: Inter-annotator agreement for all the metrics. For all the correlations presented in this table, p<0.01.
['[BOLD] Metric', '[BOLD] Spearman', '[BOLD] Pearson']
The correlation between each pair of annotations is computed and the average correlation over all the pairs is reported. Correlation is used instead of Cohen’s Kappa in order to better account for the ordinal nature of the ratings (i.e., 4 should correlate better with 5 than 1), and to maintain consistency with the evaluation of the automatic metrics. Most inter-annotator correlations are above 0.4, which indicates moderate to strong agreement. The low agreement for Understandable on PersonaChat is likely a consequence of the simple language in the dataset. Most responses are understandable, except for those requiring background knowledge (e.g., that ‘cod’ is an acronym for ‘Call of Duty’). Since the annotators have differing background knowledge, the few occasions where they fail to understand an utterance will differ, hence the lower agreement. The agreement for Overall Quality is relatively high (0.71 for Topical-Chat and 0.66 for PersonaChat) which suggests that any ambiguity in the specific dialog qualities is mitigated when the annotator is asked for an overall impression.