Are we pretraining it right? Digging deeper into visio-linguistic pretraining
Table 0.B.1: Pretraining validation losses of VisualBERT and ViLBERT. MLM points to masked language modelling object and MIM points masked image modelling objective. (Left) Losses on different same size pretraining datasets when pretrained for the same number of iterations. COCO and CC achieve the lowest and highest losses respectively. (Right) Losses on different sized subsets of CC. Adding more data consistently improves validation performance when trained for the same number of epochs. The validation set for all CC datasets are the same.
['Model', 'Same Iterations COCO', 'Same Iterations VQA 2.0 (P)', 'Same Iterations CC 10%', 'Same Epochs CC 10%', 'Same Epochs CC 50%', 'Same Epochs CC 100%']
Performance. In this section, we see how well the models perform on pretraining objectives for different pretraining datasets. It is important to note here that models sees equal number of training samples for each of the pretraining datasets and was trained for the same number of training iterations (88K). We observe that both models achieve ∼ lowest MLM loss using COCO. We hypothesize that this lower pretraining loss points to less noisy and superior quality annotations. Pretraining Dataset Size. We observe that increasing dataset size improves validation performance consistently. Here we increase dataset size for CC in subsets of 10%, 50% and 100%. Note that the number of epochs in these set of experiments is kept constant (30 Epochs) so as to not overfit smaller size datasets. We see the difference in CC 10% on left and right due to difference in the number of iterations of training (88K on left vs 12K on right), indicating longer pretraining is helpful.