Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Table 9: Performance comparison of different versions of m-RNN models on the Flickr30K dataset. All the models adopt VggNet as the image representation. See Figure 5 for details of the models.
['[EMPTY]', 'B-1', 'B-2', 'B-3', 'B-4']
To validate its efficiency, we train three different m-RNN networks: m-RNN-NoEmbInput, m-RNN-OneLayerEmb, m-RNN-EmbOneInput. “m-RNN-NoEmbInput” denotes the m-RNN model whose connection between the word embedding layer \@slowromancapii@ and the multimodal layer is cut off. Thus the multimodal layer has only two inputs: the recurrent layer and the image representation. “m-RNN-OneLayerEmb” denotes the m-RNN model whose two word embedding layers are replaced by a single 256 dimensional word-embedding layer. There are much more parameters of the word-embedding layers in the m-RNN-OneLayerEmb than those in the original m-RNN ( 256⋅M v.s. 128⋅M+128⋅256) if the dictionary size M is large. “m-RNN-EmbOneInput” denotes the m-RNN model whose connection between the word embedding layer \@slowromancapii@ and the multimodal layer is replaced by the connection between the word embedding layer \@slowromancapi@ and the multimodal layer. It verifies the effectiveness of the two word embedding layers. How to connect the vision and the language part of the model. We train three variants of m-RNN models where the image representation is inputted into the recurrent layer: m-RNN-VisualInRNN, m-RNN-VisualInRNN-both, and m -RNN-VisualInRNN-Both-Shared. For m-RNN-VisualInRNN, we only input the image representation to the word embedding layer \@slowromancapii@ while for the later two models, we input the image representation to both the multimodal layer and word embedding layer \@slowromancapii@. The weights of the two connections V(1)I, V(2)I are shared for m-RNN-VisualInRNN-Both-Shared.