Learning to Caption Images with Two-Stream Attention and Sentence Auto-Encoder
Table 2: Bleu-4 Evaluation and Reconstruction Loss for the Sentence Auto-Encoder on the Karpathy test split of MSCOCO 2014 caption dataset [21].
['Models', 'Bleu-4', '[ITALIC] Lossrecons']
Given the inputs words are not corrupted, the vanilla model outperforms the denoising one in both metrics. This is expected as the denoising model is only trained with corrupted input sequences. The loss for both the Vanilla and Denoising SAE start from a relatively high value of approximately 0.8 and 0.4 respectively, and converge to a significantly low error of 0.1 and 0.2. For a better analysis, we also compute the Bleu-4 metrics on our decoded caption against the 5 ground-truth captions. This indicates that an entire caption can be compressed in a low dimensional vector (1024) and can be successfuly reconstructed. In the next discussions, we show that they are effective to be used as a regularizer in our image captioning model. On the other hand, the Denoising model being robust to noisy summary vectors provide enough training signal to improve the image captioning decoder when initialized with hlast.