Don’t Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
Table 4: The results on the standard BLI task and BLI controlled for lexeme for the original Ruder et al. (2018)’s model (✗) and the same model trained with a morphological constraint (✓) (discussed in §4.6).
['[EMPTY]', 'Normal In vocab', 'Normal In vocab', 'Normal +OOVs', 'Normal +OOVs', 'Lexeme In vocab', 'Lexeme In vocab', 'Lexeme +OOVs', 'Lexeme +OOVs', 'Dictionary Sizes In vocab', 'Dictionary Sizes +OOVs']
We report the accuracy on the in-vocabulary pairs as well as all the pairs in the dictionary, including OOVs. As expected, compared to standard BLI this task is much easier for the models—the performance is generally high. For Slavic languages numbers remain high even in the open-vocabulary setup, which suggests that the models can generalize morphologically. On the other hand, for Romance languages we observe a visible drop in performance. We hypothesize that this difference is due to the large quantities of verbs in Romance dictionaries; in both Slavic and Romance languages verbs have substantial paradigms, often of more than 60 forms, which makes identifying the correct form more difficult. In contrast, most words in our Slavic dictionaries are nouns and adjectives with much smaller paradigms. As expected, the BLI results on unrelated languages are generally, but not uniformly, worse than those on related language pairs. The accuracy for Spanish–Polish is particularly low, at 28% (for in vocabulary pairs). We see large variation in performance across morphosyntactic categories and more and less frequent lexemes, similar to that observed for related language pairs. In particular, we observe that 2;imp;pl;v—the category difficult for Polish– Czech BLI is also among the most challenging for Polish–Spanish. However, one of the highest performing categories for Polish–Czech, 3;masc;pl;pst;v, yields much worse accuracy for Polish–Spanish. In our final experiment we demonstrate that improving morphological generalization has the potential to improve BLI results. We show that enforcing a simple, hard morphological constraint at training time can lead to performance improvements at test time—both on the standard BLI task and the controlled for lexeme BLI. We adapt the self-learning models of Artetxe et al. so that at each iteration they can align two words only if they share the same morphosyntactic category. Note that this limits the training data only to word forms present in UniMorph, as those are the only ones for which we have a gold tag. We take this as evidence that properly modelling morphology will have a role to play in BLI.