Online and Linear-Time Attention by Enforcing Monotonic Alignments
Table 4: Performance on the IWSLT 2015 English-Vietnamese TED talks for our monotonic alignment model and the baseline softmax-attention model of luong2015stanford.
['Method', 'BLEU']
Most noticeable is that the monotonic alignment model tends to focus attention later in the input sequence than the baseline softmax-attention model. We hypothesize that this is a way to compensate for non-monotonic alignments when a unidirectional encoder is used; i.e. the model has effectively learned to focus on words at the end of phrases which require reordering, at which point the unidirectional encoder has observed the whole phrase. This can be seen most clearly in the example on the right, where translating “a huge famine” to Vietnamese requires reordering (as suggested by the softmax-attention model’s alignment), so the hard monotonic alignment model focuses attention on the final word in the phrase (“famine”) while producing its translation. We suspect our model’s small decrease in BLEU compared to the baseline model may be due in part to this increased modeling burden.