All-in-One Image-Grounded Conversational Agents
Table 14: Accuracy on COCO test set when downsampling COCO during training to the same size as the Flickr30k training set, or multiples thereof. Smaller training sets are clearly helped by multi-tasking. Eventually there is enough data of the single task.
['COCO Train Size', 'Multi-Task', 'Single-Task']
The tasks we do see a performance gain in when multi-tasking, Flickr30k and Personality-Captions, tend to be smaller tasks where there is a larger related task also in the multi-tasking set, in this case COCO and Image Chat. To investigate the effects of training set size on multi-tasking transfer we thus conducted experiments to see if we observe the same effects of improvement on another dataset if we downsampled it. We thus consider adjusting the training set size of COCO to be the same size as Flickr30k, and then consider multiples of that size, and observe the change in performance with changing size. We compare single-task training on that subset to multi-task training with all other tasks and that subset. For these experiments we considered a smaller hyperparameter sweep for simplicity, with a multimodal combiner of 2 layers and sweep across different number of heads for the multi-head attention, explaining the slightly lower results. We perform early stopping on COCO. We observe for single-task training a drop from 54% accuracy to 42.1% as we go from 83k examples down to 29k. Multi-tasking with the full COCO dataset also yields the same 54% accuracy, which makes it appear that multi-tasking is not useful for generalization. However, subsampling COCO reveals a different story – the smaller the training set, the more the multi-tasking helps, with a gap of 42.1% to 49.3% in the 29k training example case. As researchers who construct new tasks often collect large scale datasets, this means multi-tasking will often have less effect than is observed in a few-shot setup.