On the Linguistic Representational Power of Neural Machine Translation Models
Table 12: Comparing bilingual and multilingual Czech↔English models across translation quality and classifying different linguistic properties. bi = bilingual model, bi* = larger bilingual model, multi = multilingual model, multi* = larger multilingual model. Larger models have twice the number word embedding and LSTM state dimensions. The larger models improve the substantially the multilingual systems, but only slightly improve the bilingual systems. Morphology = Morphological Tagging, Syntax = Syntactic Dependency Labeling, Semantics = Semantic Dependency Labeling
['[EMPTY]', 'cs-en BLEU', 'cs-en Morphology', 'cs-en Syntax', 'cs-en Semantics', 'en-cs BLEU', 'en-cs Morphology', 'en-cs Syntax', 'en-cs Semantics']
We see that the many-to-one multilingual system, i.e., *-to-English, is on par or slightly behind bilingual systems when trained with the same number of parameters as the bilingual models. In contrast, the one-to-many multilingual system, i.e., English-to- *, is significantly worse compared to its bilingual counterparts. The reason for this discrepancy could be that generation is a harder task compared to encoding, especially when translating into morphologically-rich languages: An average difference of -1.35 is observed when translating out-of-English compared to -0.13 when translating into-English. The larger multilingual models (with twice as many parameters) restored the baseline performance, in fact showing significant improvements in many cases. We also trained two of the bilingual baselines ( Czech ↔ English) by doubling the parameters. While the large multilingual system gave an improvement of +1.4 A similar pattern was observed in the opposite direction where increasing the model size gave a BLEU improvement of +1.9 in the multilingual system, but only +0.3 in the case of the bilingual system. These results show that multilingual systems benefit from other language pairs being trained in tandem and suggest that the underlying representations are richer than the ones trained using bilingual models. We now proceed to analyze these representations in light of the understudied linguistic properties. We consistently found that increasing the model size does not lead to the same improvements as we observed in the case of multilingual models. These results reinforce that multilingual NMT models learn richer representations compared to the bilingual model and benefit from the shared properties across different language pairs.