Examining Cooperation in Visual Dialog Models
Table 1: MPR [%] of each intervention experiment with p=1.0, where we intervened at each round during inference on the validation set. The difference between "None" and all other intervention rankings at the final round is shown in the final row.
['[BOLD] Intervention', 'None', 'Images', 'Captions', 'Answers', 'Questions']
We clearly observe a large discrepancy between rankings of caption interventions and the other experiments. Surprisingly, the decline in performance caused by interventions on answers and questions is less than expected, suggesting that the dialog itself is not used effectively for image identification. Interestingly replacing images with complete noise has minimal to no impact on the performance of the model. Q-BOT mainly relies on the caption provided at the beginning of the dialog to make predictions. This suggests that there is little cooperation between two bots. We note that the ranking performance of the caption interventions is improved as the dialog continues, although it never recovers completely.