Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Table 5: The Effect of Multi-head Attention: We study the effect of the number of attention heads of B-BERT on the performance of Fake-English and Russian language on XNLI data. We fix both the number of depth and number of parameters of B-BERT and vary the number of attention heads. The difference between the performance on Fake-English and Russian (\Delta) is our measure of cross-lingual ability.
['[BOLD] Parameters (in Millions)', '[BOLD] Depth', '[BOLD] Multi-head Attention', '[BOLD] XNLI  [BOLD] Fake-English', '[BOLD] XNLI  [BOLD] Russian', '[BOLD] XNLI \\Delta']
In this section, we study the effect of multi-head attention on the cross-lingual ability of B-BERT. We fix the depth and the total number of parameters – which is a function of depth and size of hidden and intermediate layers and study the performance for the different number of attention heads.