Massive vs. Curated Embeddings for Low-Resourced Languages:the Case of Yorùbá and Twi
Table 3: FastText embeddings: Spearman ρ correlation between human judgements and similarity scores on the wordSim-353 for the three datasets analysed (C1, C2 and C3). The comparison with massive fastText embeddings is shown in the top rows.
['[BOLD] Model', 'Twi  [BOLD] Vocab Size', 'Twi  [BOLD] Spearman  [ITALIC] ρ', 'Yorùbá  [BOLD] Vocab Size', 'Yorùbá  [BOLD] Spearman  [ITALIC] ρ']
For Yorùbá, both versions are available but only embeddings trained on Wikipedia are available for Twi. We consider these embeddings the result of training on what we call massively-extracted corpora. Notice that training settings for both embeddings are not exactly the same, and differences in performance might come both from corpus size/quality but also from the background model. The 294-languages version is trained using skipgram, in dimension 300, with character n-grams of length 5, a window of size 5 and 5 negatives. The 157-languages version is trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. We train our fastText systems using a skipgram model with an embedding size of 300 dimensions, context window size of 5, 10 negatives and n-grams ranging from 3 to 6 characters similarly to the pre-trained models for both languages. Best results are obtained with minimum word count of 3. Notice that pre-trained embeddings on Wikipedia show a very low correlation with humans on the similarity task for both languages (ρ=0.14) and their performance is even lower when Common Crawl is also considered (ρ=0.07 for Yorùbá). An important reason for the low performance is the limited vocabulary. The pre-trained Twi model has only 935 tokens. For Yorùbá, things are apparently better with more than 150 k tokens when both Wikipedia and Common Crawl are used but correlation is even lower. If we focus only on Wikipedia, we see that many texts are without diacritics in Yorùbá and often make use of mixed dialects and English sentences in Twi. The character-enhanced word embeddings are trained using a skipgram architecture with cluster-based embeddings and an embedding size of 300 dimensions, context window-size of 5, and 5 negative samples.