On the Learnability of Concepts With Applications to Comparing Word Embedding Algorithms
Table 6: AUC performance of word lists for each embedding algorithm used in these experiments, along with the average AUC for an embedding across all lists. Bold denotes the embedding algorithm that performs best for a given word list. Italic denotes the best performing list for each embedding algorithm.
['[BOLD] L', 'GloVe', 'word2vec', 'fastText', 'Random']
The fastText embedding algorithm is shown to have the highest performing embedding for 8 of the 10 lists that have been tested. Glove performs best on two lists, and generally performs better than word2vec overall. The word list Lrelative is shown to have the best overall performance across all three non-random embeddings, demonstrating the quality of that list. We tested the statistical significance of the performance differences observed between GloVe and fastText. We use a sample size of 10 as the difference of no pairs are equal to zero. We set our alpha to 0.01 for a one sided (right) tail test, where the test statistic Wcrit is 5. We find our resulting Wtest to be 3, which leads us to reject the null hypothesis and show that fastText outperforming GloVe is statistically significant, for the word lists that we are testing. This gives us a p-value of 0.0088.