XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Table 7: Impact of using various word embeddings for initialization on multilingual distillation. SVD, PCA, and Glove uses 300-dimensional word embeddings.
['Word Embedding', '[ITALIC] F1-score', 'Std. Dev.']
Multilingual 300d FastText embeddings bojanowski2016enriching leads to minor improvement due to 38% overlap between FastText tokens and mBERT wordpieces. English 300d Glove does much better. We experiment with dimensionality reduction techniques and find SVD to work better. Surprisingly, it leads to marginal improvement over mBERT embeddings before reduction. As expected, mBERT embeddings after fine-tuning perform better than that from pre-trained checkpoints. Multilingual word embeddings: Random initialization of word embeddings work well.