Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Table 3: Results on the XQA. The average column is the average of fr and de result.
['[EMPTY]', 'en', 'fr', 'de', 'average']
Keeping the same experimental setup as XNLI, we also evaluate Unicoder on XQA and set XLM fine-tuned on our dataset as baseline with their published code. We split 5K data from training dataset as development data to do model selection. Ablation experiments for each factor showed that removing any tasks will lead to performance drop. First, the cross-lingual paraphrase classification has least drop compared to others and removing the word recovery task hurts performance significantly. For example, in the case of XNLI, using just the Cross-lingual Language Model and Paraphrase Classification improves test accuracy on average by 0.4%. And integrating with Word Recovery model allows Unicoder to learn a better representation improves the average accuracy another 0.3%. Second, Multi-language fine-tuning is helpful to find the relation between languages, we will analyze it below. With the help of Multi-language fine-tuning, Unicoder is been improved by 1.6% of accuracy on XNLI and 3.3% on XQA.