Evaluating Visual Conversational Agents via Cooperative Human-AI Games
Table 2: Performance of Human-Alice and Qbot-Alice teams measured by MR (lower is better). We observe that AI-AI teams outperform human-AI teams.
['[BOLD] Team', '[BOLD] AliceSL', '[BOLD] AliceRL']
AI-Alice teams versus human-Alice teams. Interestingly, we observe that AI-Alice teams outperform human-Alice teams. On average, a Qbot (SL)-AliceSL team takes about 5.6 guesses to arrive at the correct secret image (as opposed to 6.86 guesses for a human-AliceSL team). Similarly, a Qbot (RL)-AliceRL team takes 4.7 guesses as opposed to a human-AliceRL team which takes 7.19 guesses. When we compare AI-AI teams (see Row 2 and 3) under different settings, we observe that teams having Qbot (RL) as the questioner outperform those with Qbot (SL). Qualitatively, we found that Qbot (SL) tends to ask repeating questions in a dialog and that questions from Qbot (RL) tend to be more visually grounded compared to Qbot (SL). Also, note that among the four teams Alice does not seem to affect performance across SL and RL.