Visual Agreement Regularized Training for Multi-Modal Machine Translation
Table 4: The results of two weighting (WT) strategies based on the soft regularization.
['Methods', 'EN→DE BLEU', 'EN→DE METEOR', 'EN→FR BLEU', 'EN→FR METEOR']
Considering that the generation of non-visual words requires little visual information from the image, we propose adaptive weighting, which aims to assign reasonable weights to visual regularization term automatically. The reason is that based on the degree of dependence of the word to be generated on the image, the proposed adaptive weighting can adaptively adjust the weight of the regularization term. The weights corresponding to visual words are larger, which introduces visual agreement regularization to leverage visual information more effectively. For non-visual words, however, a smaller weight will be assigned to the regularization term in the training objective, which prevents the presence of misleading gradient signals.