Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Table 3: F1 score on the BUCC bitext mining task.
['[BOLD] Model', '[BOLD] DE-EN', '[BOLD] FR-EN', '[BOLD] RU-EN', '[BOLD] ZH-EN', '[BOLD] Avg.']
Using mean pooling directly on mBERT / XLM-R produces low scores. XLM-R mean achieves only an F1 of 11.6. While training on English NLI and STS data improves the performance for XLM-R (XLM-R-nli-stsb), it reduces the performance for mBERT. It is unclear why mBERT mean and XLM-R mean produce vastly different scores and why training on NLI data improves the cross-lingual performance for XLM, while reducing the performance for mBERT. But in conclusion, we see that mBERT / XLM-R do not have well aligned vector spaces and training only on English data is not sufficient for cross-lingual tasks.