Enhancing Machine Translation with Dependency-Aware Self-Attention
Table 1: Test Bleu (and Ribes for En-Ja) scores on small-scale (left) and large-scale (right) data sets. Models that also require target-side syntax information are marked with {}^{\dagger}, while {}^{\Uparrow} indicates statistical significance (p<0.01) against the Transformer baseline via bootstrap re-sampling koehn-2004-statistical.
['[BOLD] Method', '[BOLD] NC11  [BOLD] En-De', '[BOLD] NC11  [BOLD] De-En', '[BOLD] WMT18  [BOLD] En-Tr', '[BOLD] WMT16  [BOLD] En-De', '[BOLD] WMT17  [BOLD] En-De', '[BOLD] WAT  [BOLD] En-Ja [B]', '[BOLD] WAT  [BOLD] En-Ja [R]']
Clearly, the base Transformer outperforms previous syntax-aware RNN-based approaches, proving it to be a strong baseline in our experiments. The table shows that the simple approach of \newcitesennrich2016linguistic does not lead to notable advantages when applied to the embeddings of the Transformer model. We also see that the multi-task approach benefits from better parameterization, but it only attains comparable performance with the baseline on most tasks. On the other hand, LISA, which embeds syntax in a self-attention head, leads to modest but consistent gains across all tasks, proving that it is also useful for NMT. Finally, Pascal outperforms all other methods, with consistent gains over the Transformer baseline independently of the source language and corpus size : It gains up to +0.9 Bleu points on most tasks and a substantial +1.75 in Ribes isozaki2010ribes, a metric with stronger correlation with human judgments than Bleu in En\leftrightarrowJa translations. On WMT17, our slim model compares favorably to other methods, achieving the highest Bleu score across all source-side syntax-aware approaches. Due to the large computational cost required to train Transformer models, we only searched hyperparameters in a small grid. In order to estimate the sensitivity of the proposed approach to hyperparameters, we trained the NC11 De-En model with the hyperparameters of the En-De one. In fact, despite being trained on the same data set, we find that more Pascal heads help when German (which has a higher syntactic complexity than English) is used as the source language.