A Revised Generative Evaluation of Visual Dialogue
Table 6: Overlap and embedding distance metrics computed for k generations against the human-annotated reference set H on the validation subset Hv. For hrea-qih-g, on average ∼6 answers are the empty string, which are excluded from the computation. Metrics marked ↑ indicate higher values are better, and those marked ↓ indicate lower values are better. When k>0, k answer generations are sampled from the model—μ,σ,andγ are the mean, standard deviation and maximum of the k scores, respectively, averaged over the dataset. Otherwise, 1 answer generation is sampled and the mean μ is shown.
['Model', 'cider↑ n=1', 'cider↑ n=2', 'cider↑ n=3', 'cider↑ n=4', 'bleu↑ n=1', 'bleu↑ n=2', 'bleu↑ n=3', 'bleu↑ n=4', 'meteor↑', 'bert  [ITALIC] L2↓', 'bert CS↑', 'FastText  [ITALIC] L2↓', 'FastText CS↑']
We show the mean μ, standard deviation σ, and maximum γ across the k scores.