Visual Agreement Regularized Training for Multi-Modal Machine Translation
Table 5: The results of visual attention distance (VAD) on the EN→DE and EN→FR translations. The symbol “↓” indicates that lower is better.
['VAD (↓) Methods', 'EN→DE  [ITALIC] visual', 'EN→DE  [ITALIC] non-visual', 'EN→FR  [ITALIC] visual', 'EN→FR  [ITALIC] non-visual']
In order to verify that our approach can effectively improve the agreement of visual attention of bidirectional models, here we introduce a new evaluation metric: visual agreement distance (VAD). The proposed VAD aims at characterizing the difference of visual attention when generating semantically equivalent words. Formally, for the given instance (v,x,y), we define VAD as: VAD=1|N(x,y)|∑(x,y)∈N(x,y)ℓ1(Ax→v,Ay→v) (21) where ℓ1(⋅,⋅) denotes ℓ1 distance. N(x,y) denoted mutually aligned word pairs in (x,y), which is defined as: N(x,y)={(x,y)|x∈x,y∈y,x↔y} (22) where x↔y represents that x and y are uniquely aligned with each other. The large VAD means that the model suffers from the poor visual agreement. The results show that our approach can effectively improve the agreement of visual attention to the image of bidirectional models in the generation of semantically equivalent word pairs. In addition, such improvement is more obvious for visual word pairs compared with non-visual word pairs (e.g. 0.44 decline for visual v.s. 0.22 decline for non-visual on EN→DE). As analyzed in the previous section, the generation of visual words exhibits more dependence on the image. Therefore, the adaptive weighting will assign a greater weight to the corresponding regularization, leading to a more significant improvement.