Generating Wikipedia by Summarizing Long Sequences
Table 6: Side-by-side for two models pair with large automatic metric gaps
['[BOLD] Model A', '[BOLD] Model B', '[BOLD] ROUGE-L A', '[BOLD] ROUGE-L B', '\\# prefer B\\# prefer A']
Human Evaluation - side-by-side preference We validated our chosen metrics correlate with human preference by conducting two side-by-side human evaluation experiments, comparing models with large gaps in perplexity/ROUGE. Which side a model appears on is randomized per example and rater. we had 3 raters score 25 examples each and computed the ratio of ratings preferring one model over the other.