The Influence of Down-Sampling Strategieson SVD Word Embedding Stability
Table 1: Performance of different algorithms and down-sampling strategies with models trained on corpora with and without subsampling. Bold values are best or not significantly different by independent t-tests (with p<0.05).
['Corpus', 'Algorithm', 'Down- sampling', 'Word Similarity MEN', 'Word Similarity MTurk', 'Word Similarity SimLex', 'Word Similarity WS-353', 'Analogy Google', 'Analogy MSR', 'Stability']
Measured with j@10 metric (higher is better). Accuracy differences between test sets are in line with prior observations and general performance on WIKI is roughly in-line with the data reported in Levy et al. The strongest counterexample is a combination of probabilistic down-sampling for df and weighting for ff which lead to small, yet significant improvements in the MEN (0.703±0.001) and MTurk (0.568±0.015) similarity tasks (cf. However, other accuracy tasks showed no improvements and the stability of this approach (0.475±0.001) was far closer to SVD\textscPPMI with fully probabilistic down-sampling than to the perfect stability of SVD\textscwPPMI. Analyses on individual corpora would have resulted in insufficient samples given the pre-conditions of our tests.