Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Table 11: Premise and Hypothesis in different language: Using XNLI test set, we construct textual entailment data with premise and hypothesis in different languages. The column A-B (e.g. enfake-target) refers to test data with premise in language A (enfake) and hypothesis in language B (target). We always train on Fake-English and report test accuracy.
['[BOLD] B-BERT', '[BOLD] Target', '[BOLD] Premise Language – Hypothesis Language (XNLI)  [BOLD] enfake-target', '[BOLD] Premise Language – Hypothesis Language (XNLI)  [BOLD] target-enfake', '[BOLD] Premise Language – Hypothesis Language (XNLI)  [BOLD] enfake-enfake', '[BOLD] Premise Language – Hypothesis Language (XNLI)  [BOLD] target-target']
We observe a drastic drop in the entailment performance of B-BERT when the premise and hypothesis are in different languages. (This data was created using XNLI where in the original form the languages contain the same premise and hypothesis pair). One of the possible explanations could be that BERT is learning to make textual entailment decisions by matching words or phrases in the premise to those in the hypothesis. This question, too, is left as a future direction.