Don’t Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
Table 2: The sizes of our morphologically complete dictionaries for Slavic and Romance language families. We present the sizes for 20 base dictionaries. We further split those to obtain 40 train, development and test dictionaries—one for each mapping direction to ensure the correct source language lemma separation.
['[BOLD] Slavic', 'Czech', 'Russian', 'Slovak', 'Ukrainian', '[BOLD] Romance', 'Spanish', 'Catalan', 'Portuguese', 'Italian']
For each language pair (L1,L2) we first generated lemma translation pairs by mapping all L1 lemmata to all L2 lemmata for each synset that appeared in both L1 and L2 WordNets. We then filtered out the pairs which contained lemmata not present in UniMorph and generated inflected entries from the remaining pairs: one entry for each tag that appears in the UniMorph paradigms of both lemmata. The sizes of dictionaries vary across different language pairs and so does the POS distribution. In particular, while Slavic dictionaries are dominated by nouns and adjectives, verbs constitute the majority of pairs in Romance dictionaries. In our split, the train dictionary contains 60% of all lemmata, while the development and test dictionaries each have 20% of the lemmata.