Agreement-based Joint Training for Bidirectional Attention-based Neural Machine Translation
Table 4: Comparison of independent and joint training in terms of average attention entropy (see Eq. (15)) on Chinese-to-English translation.
['Word', 'Type', 'Freq.', 'Indep.', 'Joint']
We find that the entropy generally goes downs with the decrease of word frequencies, which suggests that frequent target words tend to gain attention from multiple source words. Apparently, joint training leads to more concentrated attention than independent training. The gap seems to increase with the decrease of word frequencies.