Improving Sparse Word Representations with Distributional Inference for Semantic Composition
Table 3: Effect of the magnitude of the shift parameter k in SPPMI on the word similarity tasks. Boldface means best performance per dateset.
['Apt [BOLD] s', '[BOLD] MEN  [ITALIC] without DI', '[BOLD] MEN  [ITALIC] with DI', '[BOLD] SimLex-999  [ITALIC] without DI', '[BOLD] SimLex-999  [ITALIC] with DI', '[BOLD] WordSim-353 (rel)  [ITALIC] without DI', '[BOLD] WordSim-353 (rel)  [ITALIC] with DI', '[BOLD] WordSim-353 (sub)  [ITALIC] without DI', '[BOLD] WordSim-353 (sub)  [ITALIC] with DI']
For the Apt model, a value of k=40 performs best (except for SimLex-999, where smaller shifts give better results), with a performance drop-off for larger shifts. In our experiments we find that a shift of k=1 results in top performance for the untyped vector space model. It appears that shifting the PPMI scores in the Apt model has the effect of cleaning the vectors from noisy PPMI artefacts, which reinforces the predominant sense, while other senses get suppressed. Subsequently, this results in a cleaner neighbourhood around the word vector, dominated by a single sense. This explains why distributional inference slightly degrades performance for smaller values of k.