Word Embeddings and Stability
Table 1.5: Scores of our word2vec models (skip-gram) on the word analogy tasks for different languages compared to the results published by bojanowski2016. The results of this work, noted in the rightmost column, state the mean μ, standard deviation σ, and highest score of 128 runs on independently shuffled corpora.
['[BOLD] Language', '[BOLD] Bojanowski (2016)', '[BOLD] This Work  [ITALIC] μ', '[BOLD] This Work  [ITALIC] σ', '[BOLD] This Work Best']
Following grave-etal-2018-learning, the vocabulary of each model was restricted to the 200,000 most frequent words from the training data before evaluating the model on the word analogy dataset. This means, that a fraction of the questions of the analogy tasks is not answered, as they contain out-of-vocabulary words. Therefore, to compare the reported scores one also needs to take the coverage, i.e. the percentage of answered questions, into account. bojanowski2016 did not publish these numbers.