Effects of Language Relatedness for Cross-lingual Transfer Learning in Character-Based Language Models
Table 2: The table reports NNLM’s test set perplexity for Finnish and Swedish using different cross-lingual initializations. For Finnish, English and Estonian are used as the source languages for pretraining. For Swedish, we use Danish, English and Norwegian as source languages. The best results in each category are marked in boldface.
['[BOLD] Finnish Test Set Perplexity Fi0 (baseline)', '[BOLD] Finnish Test Set Perplexity  [BOLD] 3788', '[BOLD] Finnish Test Set Perplexity  [BOLD] 3788', '[BOLD] Finnish Test Set Perplexity  [BOLD] 3788', '[BOLD] Finnish Test Set Perplexity  [BOLD] 3788']
When using related source languages — like Estonian for Finnish, or Danish and Norwegian for Swedish — to pretrain the models, we obtain better perplexity than the baseline and when English (the unrelated source language) is used, which leads to a worse perplexity for all ls. Using related source languages, the pretrained target LMs outperform the baseline results for most values of l but, notably, when initialized with configurations l=1,4 of the source model. Here, we note that Finnish perplexity values are large due to long words and the domain mismatch between the training (books, newspaper articles and journals) and test (broadcast news) sets.