Image Captioning with Very Scarce Supervised Data:Adversarial Semi-Supervised Learning Approach
Table 3: Evaluation of our method with different backbone architectures. All models are reproduced and trained with the cross entropy loss. By adding Unlabeled-COCO images, our training method was applied in a semi-supervised way, which shows consistent improvement in all the metrics.
['[EMPTY]', 'BLEU1', 'BLEU2', 'BLEU3', 'BLEU4', 'ROUGE-L', 'SPICE', 'METEOR', 'CIDEr']
We implement our neural networks by PyTorch library Paszke et al. We use ResNet101 He et al. Each region is represented as 2,048-dimensional output from the last residual block of this network. For NIC model Vinyals et al. et al. For Up-Down Anderson et al. Ren et al. object dataset, thereby we extract visual features for 10 to 100 regions. To further validate our method in the semi-supervised setup, we compare different backbone architectures Anderson et al. We use the same data setup with the above, but we replace CNN (F) and LSTM (H) from our framework with the image encoder and the caption decoder from their image captioning models. Then, these models can be trained in our framework as it is by alternating between the discriminator update and pseudo-labeling. Training with the additional Unlabeled-COCO data via our training scheme consistently improves all baselines in all metrics.