Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Table 4: The Effect of Depth of B-BERT Architecture: We use Fake-English and Russian B-BERT and study the effect of depth of B-BERT towards XNLI. We vary the depth and fix both the number of attention heads and the number of parameters – the size of hidden and intermediate units are changed so that the total number of parameters remains almost the same. We train only on Fake-English and test on both Fake-English and Russian and report their test accuracy. The difference between the performance on Fake-English and Russian(\Delta) is our measure of cross-lingual ability (lesser the difference, better the cross-lingual ability).
['[BOLD] Parameters (in Millions)', '[BOLD] Depth', '[BOLD] Multi-head Attention', '[BOLD] XNLI  [BOLD] Fake-English', '[BOLD] XNLI  [BOLD] Russian', '[BOLD] XNLI \\Delta']
We can also see a strong correlation between performance on English and cross-lingual ability (\Delta), which further supports our assumption that the ability to extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness.