A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Table 2: Best BLEU-1 score of mean difference shift on Unsupervised Token Translation with best \alpha is selected from [1.0,2.0,3.0], and the best layer is selected from [1,12].
['[EMPTY]', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}de', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}fr', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}ur', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}sw', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}zh', 'en\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}el', 'de\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en', 'fr\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en', 'ur\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en', 'sw\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en', 'zh\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en', 'el\\mathrel{\\hbox{\\rule[-0.2pt]{3.0pt}{0.4pt}}\\mskip-4.0mu \\hbox{\\char 41}}en']
We computed the BLEU-1 score using ground truth reference translation of and the convert rate each sentence in XNLI test-set. \alpha and different layers. We observed the influences of weight increase on convert rate were monotonic increase, there are more converted tokens as weight increasing. It indicates that the last few layers may be better for disentangling language-specific representations, which is consistent with the observation in the literature that the last few layers contain more language-specific information for predicting masked words (pires:19).