Syntactic Interchangeability in Word Embedding Models
Table 1: Analysis of interchangeability (by same-POS) in word similarity and relatedness benchmarks. Δwin=2→15(%) is the relative change, in percents, of the model’s performance (by Spearman correlation) when going from window size 2 to window size 15, for the CBOW and SGNS algorithms (§2.3). Related and Unrelated are the top and bottom 30% of the pairs, by benchmark score, respectively. P-value is calculated using the hypergeometric test, comparing the enrichment of interchangeable pairs within related pairs, with a background of all related and unrelated pairs (§3.1).
['[BOLD] Benchmark', '[BOLD] Size', 'Δwin=2→15(%)  [BOLD] CBOW', 'Δwin=2→15(%)  [BOLD] SGNS', '# [BOLD]  Related  [BOLD] All', '# [BOLD]  Related  [BOLD] Same-POS', '# [BOLD]  Unrelated  [BOLD] All', '# [BOLD]  Unrelated  [BOLD] Same-POS', '[BOLD] p-value']
S2SS1SSS0Px1 Benchmarks. We use the following benchmarks: WordSim-353 Finkelstein et al. Agirre et al. MTurk-287 Radinsky et al. SimVerb-3500 Gerz et al. Interestingly, CBOW exhibits a positive correlation of window size with model’s performance for all benchmarks but SimLex999, while performance for SGNS barely changes with window size, except for SimLex999 and SimVerb3500, where we see a strong negative correlation.