Assessing the Bilingual Knowledge Learned by Neural Machine Translation Models
Table 1: Comparison of the phrase table extracted from the full training data (“Full”) and NMT models (“NMT”). “All” denotes the whole phrase table, “Shared” denotes the intersection of two tables, and “Non-shared” denotes the complement. Note that the probabilities of “Shared” phrases are different for the two tables.
['[BOLD] Phrase Table', '[BOLD] Shared  [ITALIC] Size', '[BOLD] Shared  [ITALIC] BLEU', '[BOLD] Non-Shared  [ITALIC] Size', '[BOLD] Non-Shared  [ITALIC] BLEU', '[BOLD] All  [ITALIC] Size', '[BOLD] All  [ITALIC] BLEU']
(17.90 vs. 17.91), but with only a half of phrases (9.0M vs. 17.5M). In addition, NMT provides a better probability estimation for the distilled phrases (“Shared”, 17.90 vs. 17.32). In the “Non-Shared” table, 78.2% of the phrase pairs share the same source phrase with the “Shared” table, of which 83.2% have a lower translation probability. The results empirically confirm our hypothesis that NMT models distill the bilingual knowledge by discarding those low-quality phrase pairs.