Probing for Semantic Classes:Diagnosing the Meaning Content of Word Embeddings
Table 6: Similarity and analogy results of our word embeddings on a set of datasets Jastrzebski et al. (2017). The table shows the Spearmans correlation between the model’s similarities and human judgments. Size is the dimensionality of the embeddings. Except for RW dataset, results improve by increasing embeddings size.
['model', 'size', 'MEN', 'MTurk', 'RW', 'SimLex999', 'WS353', 'Google', 'MSR']
Higher dimensional embeddings perform better for MLP and LR, but worse for KNN. We do further analysis by counting the number k of unique S-classes in the top 5 nearest neighbors for word embeddings; k is 1.42 times larger for embeddings of dimensionality 400 than 200. Thus, more dimensions results in more diverse neighborhoods and more randomness. We explain this by the increased degrees of freedom in a higher dimensional space: idiosyncratic properties of words can also be represented given higher capacity and so similarity in the space is more influenced by idiosyncracies, not by general properties like semantic classes.