CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
Table 1: Results for state-of-the-art models on the CompGuessWhat?! suite of evaluation tasks. We assess model quality in terms of gameplay accuracy, the attribute prediction quality, measured in terms of F1 for the abstract (A-F1), situated (S-F1), abstract+situated (AS-F1) and location (L-F1) prediction scenario, as well as zero-shot learning gameplay. The final score GroLLA is a macro-average of the individual scores. We use the models GloVe, ResNet and GDSE-*-text only as a baseline for the attribute prediction tasks.
['[EMPTY]', '[ITALIC]  [BOLD] Gameplay  [BOLD] Accuracy', '[ITALIC]  [BOLD] Attribute Prediction  [BOLD] A-F1', '[ITALIC]  [BOLD] Attribute Prediction  [BOLD] S-F1', '[ITALIC]  [BOLD] Attribute Prediction  [BOLD] AS-F1', '[ITALIC]  [BOLD] Attribute Prediction  [BOLD] L-F1', '[ITALIC]  [BOLD] Zero-shot Gameplay  [BOLD] ND-Acc', '[ITALIC]  [BOLD] Zero-shot Gameplay  [BOLD] OD-Acc', '[ITALIC]  [BOLD] GroLLA']
Guesser accuracy We evaluate the GDSE and DeVries models in gameplay mode using the set of reference games provided in CompGuessWhat?!. dataset de Vries et al. With the AP task, we try to answer the following question: “Do the representations associated with the target object encoding provide useful information that can be exploited to predict the object attributes correctly?” We assume that, due to the nature of the CompGuessWhat?! games, the final dialogue state representation should encode relevant features of the target object. So, a high gameplay accuracy should correlate with a high AP score. test games. As the average best model performance was only 44.27, far from ceiling, our hypothesis is only partially supported. In particular, the models having the highest guesser accuracy, GDSE-CL and GDSE-SL, seem to learn better representations than unimodal baselines GloVe and ResNet, confirming the importance of multi-modal training for this task. There is also a gap in performance between the GDSE and DeVries models. This might be related to the multi-task learning strategy used by GDSE models that favours the emergence of more expressive representations than the ones learned by DeVries models which are trained in isolation. By comparing the enhanced versions GDSE-CL and DeVries-RL with the less sophisticated ones, GDSE-SL and DeVries-SL, respectively, we observe that, despite their higher guesser accuracy, these models do not have any advantage in terms of the AP task. We believe that this is because the Collaborative training strategy (for GDSE-CL) and Reinforcement Learning (for DeVries-RL) are optimising end-goal performance while sacrificing the expressiveness of the representations. Finding a way to encode task-specific representations and generalise them to learn abstract representations becomes an important research direction to improve on this task. would result in a successful gameplay.