Supervised Attentions for Neural Machine Translation
Table 1: Single system results in terms of (Ter-Bleu)/2 (T-b, the lower the better) on 5 million Chinese to English training set. Bp denotes the brevity penalty. NMT results are on a large vocabulary (300k) and with UNK replaced. The second column shows different alignments (Zh → En (one direction), GDFA (“grow-diag-final-and”), and MaxEnt [Ittycheriah and Roukos2005]. A, T, and J mean optimize alignment only, translation only, and jointly. Gau. denotes the smoothed transformation.
['single system', 'single system', 'single system', 'MT06', 'MT06', 'MT06', 'MT08 News', 'MT08 News', 'MT08 News', 'MT08 Web', 'MT08 Web', 'MT08 Web', 'avg.']
The syntax-based statistical machine translation model achieves an average (Ter-Bleu)/2 of 13.36 on three test sets. The Cov. LVNMT system achieves an average (Ter-Bleu)/2 of 14.24, which is about 0.9 points worse than Tree-to-string SMT system. Please note that all systems are single systems. It is highly possible that ensemble of NMT systems with different random seeds can lead to better results over SMT. First, with the same alignment, J joint optimization works best than other optimization strategies (lines 3 to 6). Unfortunately, breaking down the network into two separate parts (A and T) and optimizing them separately do not help (lines 3 to 5). We have to conduct joint optimization J in order to get a comparable or better result (lines 3, 5 and 6) over the baseline system.