Neural Polysynthetic Language Modelling
Table 4.9: YiSi-1 scores (higher is better) computed using ess or esu BPE 5000 embeddings built by word2vec (w2v) for experiments on multilingual neural machine translation. The baseline is the original Transformer baseline for each language pair. Multilingual is the single multilingual system (trained on Inuktitut and Yupik data), and the remaining two columns show that system fine-tuned on a particular variety of Yupik.
['[EMPTY]', 'Baseline', 'Multilingual', 'ess-Ad. Multi.', 'esu-Ad. Multi.']
We train joint BPE (vocabulary 5000) on Inuktitut, St. Lawrence Island Yupik, and Central Alaskan Yup’ik, downsampling the Inuktitut and upsampling St. Lawrence Island Yupik to match the size of Central Alaskan Yup’ik. We prepend a language tag (e.g. “ ”) to each source and target sentence in the three sub-corpora. Next we train a Transformer model (our “multilingual baseline”) on the concatenation of all available training data (with no sampling, 3 layer encoder, 3 layer decoder, 512 embedding size, early stopping on perplexity of the concatenated development data). For St. Lawrence Island Yupik and Central Alaskan Yup’ik, we then fine-tune the multilingual baseline on all language-specific training data (with early stopping based on perplexity on the language-specific development data). As expected, fine-tuning on language specific data boosts performance on that particular language (while the output on the other language appears to exhibit catastrophic forgetting (Kirkpatrick3521)), giving us our best performance. However, with BLEU scores in the single digits, it is clear that there is still a long way to go before the MT output may be genuinely useful (e.g. in post-editing or interactive translation) for these low-resource languages.