A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Table 4: Experiment results of applying mean difference shift to m-BERT on XNLI test-set. Shifting weights are selected according to the best results on XNLI dev-set.
['[EMPTY]', 'en', 'de', 'es', 'ar', 'el', 'fr', 'hi', 'ru', 'th', 'tr', 'vi', 'zh', 'avg w/o en']
When applying MDS, we used different shifting weights \alpha for different languages considering the sensitivity may differ among languages. We got a slight improvement on the testing sets from most of the languages, except Spanish (es) and French (fr). The results show that by simply using MDS on the XNLI dataset, the language-specific information can be eliminated to some extent, and the modified representations can preserve more semantic information instead of mixed with language-specific information.