Can Multilingual Language Models Transfer to an Unseen Dialect?A Case Study on North African Arabizi
Table 1: Cross-Lingual performance averaged on 5 seeds on the Narabizi test set. Baselines are described in Section 3.
['[ITALIC] Source', 'mBERT+Task POS', 'mBERT+Task UAS', 'mBERT+MLM+Task POS', 'mBERT+MLM+Task UAS']
To grasp the complexity of Narabizi, we run some preliminary experiments. Competitive taggers perform on datasets of similar size above 90%. StanfordNLP only reaches 84.20% on our data for POS tagging and 52.84% for parsing, as measured by the unlabeled attachment score (UAS; cf. We report the accuracy for POS tagging and the Unlabeled Attachment Score (UAS) for parsing. For both POS tagging and parsing, mBERT+Task performs outperforms the baselines by a large margin when the source is Maltese, French and English. This shows that mBERT is able to transfer to Narabizi even without having been trained on any Narabizi tokens at any stage of the training process. We hypothesize that the high level of transfer when the source is French is due to the high code-mixing proportion of Narabizi. the performance of the model with respect to the code-mixing ratio. We split the dataset into four buckets of around 25% of the full dataset, according to the ratio of native Narabizi tokens in each sentence (between less than 60% to stricly 100%) as opposed to French tokens. We compare French and Maltese as source languages. We confirm our intuition that code-mixing explains the good performance of the model trained on French. Indeed, on sentences that have 100% Narabizi tokens, mBERT+Task trained on French performs poorly (cf. fig. On the other side, for sentences that include at least 40% of French tokens, scores reach 54% (cf. (A)) for POS tagging and 47% for parsing (cf. (I)). Moreover, for French, mBERT+MLM+Task leads to an impressive 21.2% error reduction compared to mBERT+Task for POS tagging (33.12 vs. 47.32) and an 8.5% error reduction for parsing (cf. We observe in Fig. (B) and (K)) that this improvement mostly comes from a better accuracy on Narabizi tokens. Interestingly, we observe that unsupervised fine-tuning leads to the closing of the gap between the performance of the models tuned on French and Maltese on native Narabizi tokens (+15: (B)-(E) vs. +2.4: (B)-(C) for POS tagging and +5.6: (K)-(L) vs. +2.2: (J)-(K) for parsing). This demonstrates the capacity of unsupervised fine-tuning to close lexical mismatch between distant languages such as native Narabizi and French.