XGPT: Cross-modal Generative Pre-Training for Image Captioning
Table 4: Comparison of two masking methods on COCO Captions.
['[EMPTY]', 'C', 'B@4', 'M', 'S']
Comparing with one-stage pre-training, we find that each task combination pre-trained after the second stage (Row 6-8) gains approximately +2 on CIDEr. This indicates that two-stage pre-training with in-domain data enables the model to adapt to the downstream data better than only using out-of-domain pre-training. Combining all three tasks leads to the highest score on all metrics. We use this as the optimal pre-training setting for further experiments. How to Efficiently Mask? We conduct further experiments to compare two masking strategies for IDA: (1) Multi [MASK]: replace tokens in the sampled fragment with exactly the same number of [MASK] tokens, (2) Single [MASK]: replace the fragment with a single [MASK] token. Single [MASK] is obviously better than Multi [MASK]. Multi [MASK] provides the decoder with full position information of masked token, which reduced the difficulty for the decoder to predict correct words, thus, gives a less well performance (-0.3 on CIDEr). Therefore, we use the single [MASK] as the optimal pre-training setting.