On the Linguistic Representational Power of Neural Machine Translation Models
Table 16: Results on syntactic and semantic tagging and labeling with representations obtained from char-based models trained with an extra layer.
['[EMPTY]', 'cs-en Syn Dep', 'cs-en Sem Dep', 'en-de Syn Dep', 'en-de Sem Dep', 'en-de Sem tags']
A possible confound is that character-based models start from a lower linguistic level compared to word or subword models and may require more depth to learn long-range dependencies. To verify this, we trained 3-layered character models for Czech-to-English and English-to-German. We extracted feature representations and trained classifiers to predict syntactic dependency labels. Our results show that using an additional layer does improve the prediction accuracy, giving the same result as subword segmentation (Morfessor) in the case of Czech-to-English, but still worse in the case of English-to-Czech (see In order to probe whether character-based models require additional depth in the network to capture the same amount of information, we carried out further experiments training 3-layered character models for Czech-to-English and English-to-German. We extracted feature representations and trained classifiers to predict syntactic dependency labels.