Massive vs. Curated Embeddings for Low-Resourced Languages:the Case of Yorùbá and Twi
Table 5: NER F1 score on Global Voices Yorùbá corpus after fine-tuning BERT for 10 epochs. Mean F1-score computed after 5 runs
['[BOLD] Embedding Type', '[BOLD] DATE', '[BOLD] LOC', '[BOLD] ORG', '[BOLD] PER', '[BOLD] F1-score']
As expected, the fine-tuning of the pre-trained BERT on the Yorùbá corpus in the two configurations generates better representations than the base model. The fine-tuned BERT model with only Yorùbá vocabulary further increases by 4% in F1 score than the BERT model that uses the multilingual vocabulary. Although we do not have enough data to train BERT from scratch, we observe that fine-tuning BERT on a limited amount of monolingual data of a low-resource language helps to improve the quality of the embeddings.