All-in-One Image-Grounded Conversational Agents
Table 7: Comparison of with and without (w/o) the multimodal combiner (MMC) as part of our TransResNet architecture, for COCO, Flickr30k (Fl30k) and Image Chat (IC), using either ResNeXt-IG-3.5B (ResNeXt-IG) features alone or in combination with Faster R-CNN features. The MMC provides gains in all cases.
['ResNeXt-IG-3.5B', 'w/o MMC', 'COCO 48.8', 'Fl30k 71.8', 'IC 55.2']
Multimodal Combiner We next assess the impact of the multimodal combiner module in our architecture; we first analyze the non-attentive version. We compare these alternatives on three tasks: COCO, Flickr30k and Image Chat, and examine performance both for our best performing combination features as well as for ResNeXt-IG-3.5B alone. We see that without this component of the architecture, the model can still give somewhat reasonable performance. However, by combining modalities with a Transformer architecture we do see improvements across all tasks. The MMC module takes as input a sequence-based representation of the context history (token-level representation). We also experimented with giving the mean sequence representation as input to the MMC instead, which gave worse results (5% regression on IC). We thus report subsequent experiments using the full combiner using sequence-based inputs.