Meshed-Memory Transformer for Image Captioning
Table 1: Ablation study and comparison with Transformer-based alternatives. All results are reported after the REINFORCE optimization stage.
['[EMPTY]', 'B-1', 'B-4', 'M', 'R', 'C', 'S']
Firstly, we investigate the impact of the number of encoding and decoding layers on captioning performance. Varying the number of layers, we observe a significant increase in performance when using three encoding and three decoding layers, which leads to 123.6 CIDEr. We hypothesize that this is due to the reduced training set size and to the lower semantic complexities of sentences in captioning with respect to those of language understanding tasks. Following this finding, all subsequent experiments will use three layers. Attention on Attention baseline. We also evaluate a recent proposal that can be straightforwardly applied to the Transformer as an alternative to standard dot-product attention. Specifically, we evaluate the addition of the “Attention on Attention” (AoA) approach [huang2019attention] to the attentive layers, both in the encoder and in the decoder. Noticeably, in [huang2019attention] this has been done with a Recurrent language model with attention, but the approach is sufficiently general to be applied to any attention stage. In this case, the result of dot-product attention is concatenated with the initial query and fed to two fully connected layers to obtain an information vector and a sigmoidal attention gate, then the two vectors are multiplied together. The final result is used as an alternative to the standard dot-product attention. Meshed Connectivity. We then evaluate the role of the meshed connections between encoder and decoder layers. Using this 1-to-1 connectivity schema already brings an improvement with respect to using the output of the last encoder layer as in the standard Transformer (123.6 CIDEr vs 129.2 CIDEr), thus confirming that exploiting a multi-level encoding of image regions is beneficial. When we instead use our meshed connectivity schema, that exploits relationships encoded at all levels and weights them with a sigmoid gating, we observe a further performance improvement, from 129.2 CIDEr to 131.2 CIDEr. This amounts to a total improvement of 7.6 points with respect to the standard Transformer. Also, the result of our full model is superior to that obtained using the AoA. Role of persistent memory. We evaluate the role of memory vectors in both the 1-to-1 configuration and in the final configuration with meshed connections. Further experiments on the number of memory vectors can be found in the supplementary material.