Inducing Grammars with and for Neural Machine Translation
Table 2: Results for translating En↔De, En↔Ru, and Ru→Ar. Statistical significances are marked as †p<0.05 and ‡p<0.01 when compared against the baselines and △/▲ when compared against the FA-NMT (no-shared). The results indicate the strength of our proposed shared-attention for NMT.
['Model', 'Shared', 'De→En  [ITALIC] wmt16', 'De→En  [ITALIC] wmt17', 'Ru→En  [ITALIC] wmt17', 'En→De  [ITALIC] wmt16', 'En→De  [ITALIC] wmt17', 'En→Ru  [ITALIC] wmt17', 'Ru→Ar  [ITALIC] un-test']
We test statistical significance using bootstrap resampling (riezler-maxwell:2005:MTSumm). Statistical significances are marked as †p<0.05 and ‡p<0.01 when compared against the baselines. Additionally, we also report statistical significances △p<0.05 and ▲p<0.01 when comparing against the FA-NMT models that have two separate attention layers from the decoder to the encoder. Overall, the SA-NMT (shared) model performs the best gaining more than 0.5 BLEU De→En on wmt16, up to 0.82 BLEU on En→De wmt17 and 0.64 BLEU En→Ru direction over a competitive NMT baseline. The gain of the SA-NMT model on Ru→Ar is small (0.45 BLEU) but significant. The results show that structured self attention is useful when translating from English to languages that have long-distance dependencies and complex morphological agreements. We also see that the gain is marginal compared to self-attention models (FA-NMT-shared) and not significant. Within FA-NMT models, sharing attention is helpful. Our results also confirm the advantage of having two separate sets of annotations in the encoder when modeling syntax. The hard structured self attention model (SA-NMT-hard) performs comparably to the baseline.