Visual Dialog
Table 4: Human-machine performance comparison on VisDial v0.5, measured by mean reciprocal rank (MRR), recall@k for k={1,5} and mean rank. Note that higher is better for MRR and recall@k, while lower is better for mean rank.
['[EMPTY]', '[BOLD] Model', '[BOLD] MRR', '[BOLD] R@1', '[BOLD] R@5', '[BOLD] Mean']
Results of this study are shown in the top-half of Tab. We find that without access to the image, humans perform better when they have access to dialog history – compare the Human-QH row to Human-Q (R@1 of 30.31 vs. 25.10). As perhaps expected, this gap narrows down when humans have access to the image – compare Human-QIH to Human-QI (R@1 of 48.03 vs. 46.12). Tab. We can see that, as expected, humans with full information (i.e. Human-QIH) perform the best with a large gap in human and machine performance (compare R@5: Human-QIH 83.76% vs. MN-QIH-D 69.39%). This gap is even larger when compared to generative models, which unlike the discriminative models are not actively trying to exploit the biases in the answer candidates (compare R@5: Human-QIH 83.76% vs. HREA-QIH-G 61.61%).