CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
Table 5: Full set of attribute prediction task metrics. We evaluate F1, Precision and Recall for all the tasks. All the metrics are computed as macro-average.
['[BOLD] Models', '[ITALIC]  [BOLD] Abstract  [BOLD] F1', '[ITALIC]  [BOLD] Abstract  [BOLD] Precision', '[ITALIC]  [BOLD] Abstract  [BOLD] Recall', '[ITALIC]  [BOLD] Situated-only  [BOLD] F1', '[ITALIC]  [BOLD] Situated-only  [BOLD] Precision', '[ITALIC]  [BOLD] Situated-only  [BOLD] Recall', '[ITALIC]  [BOLD] Abstract+situated  [BOLD] F1', '[ITALIC]  [BOLD] Abstract+situated  [BOLD] Precision', '[ITALIC]  [BOLD] Abstract+situated  [BOLD] Recall', '[ITALIC]  [BOLD] Location  [BOLD] F1', '[ITALIC]  [BOLD] Location  [BOLD] Precision', '[ITALIC]  [BOLD] Location  [BOLD] Recall']
We consider the CompGuessWhat?! splits as reference for our experimental evaluation. We generate an hidden state for every successful dialogue and we use the classifier Φ to predict the target object attributes. We train the classifier Φ by minimising the binary cross-entropy loss computed between the model prediction and the reference set of attributes. To prevent overfitting, we perform early stopping on the validation set using the multi-label F1-measure (with threshold 0.75) as reference metric and we apply a learning rate scheduler to gradually reduce the learning rate. The model training has been implemented using AllenNLP Gardner et al.