On the Linguistic Representational Power of Neural Machine Translation Models
Table 17: CCG tagging accuracy using features from the k-th encoding layer of 4-layered English-to-* NMT models trained with different target languages (German, Czech, French, and Spanish).
['[ITALIC] k', 'de', 'cs', 'fr', 'es']
We previously found that morphology is predominantly being captured in layer 1 of the NMT models. We now repeat the experiments for syntactic dependencies. We found that representations from layer 4 performed better than representations from lower layers except for the French encoder, where layer 3 performs better. These results are consistent with the syntactic dependency results. We repeated these experiments with the multi-parallel UN corpus by training English-to-{French, Arabic, Spanish, Russian, and English} bilingual models. Comparing successive layers (for example, comparing layer 2 versus layer 3), in the majority of the cases, the higher layer performed statistically significantly better than the lower one (ρ < 0.01), according to the approximate randomization test sigf06. Similar to the results on morphological tagging, a combination of all layers achieved the best results. This implies that although syntax is mainly learned at higher layers, syntactic information is at least partly distributed across the network. Along with the syntactic dependency labeling task, we found higher layers to give better classifier accuracy also in the CCG tagging task.