Word Embeddings and Stability
Table 2.1: Lowest and highest scores on the word analogy tasks for different languages observed in 128 subsequent runs of fastText trained on independently shuffled Wikipedia corpora as outlined in Section 1.2.
['[BOLD] Language', '[BOLD] Lowest Score', '[BOLD] Highest Score', '[BOLD] Rel. Difference']
We could reproduce the findings of hellrich-hahn-2016-assessment; hellrich-hahn-2016-bad; hellrich-hahn-2017-fool; hellrich-etal-2019-influence; antoniak2018; Chugh2018StabilityOW; wendlandt2018; pierrejean-tanguy-2018-predicting, that the training processes of prediction-, as well as count-based non-contextualized word embeddings exhibit significant instability. The extent of this instability, that we found even for comparably large corpora might still come as a surprise to the reader. This supports the case, that every time a score on a word analogy dataset – or any task that depends on word embeddings for that matter –- is published, the results of at least five independent runs should be reported. The current practice in research is to provide a single score, without information on its variance; our data indicates that this is insufficient.