1 INTRODUCTION
Table 3: Performance comparison between dimensionality reduction with stochastic low-rank approximation (SLRA) and Hellinger PCA (HPCA). A symmetric context of five or ten words with scenarios (1) and (4) have been used. The best three results for each dataset are in bold, and the best is underlined. Spearman rank correlation is reported on word similarity tasks. Accuracy is reported on word analogy tasks.
['Dimension', '[BOLD] SLRA 100', '[BOLD] SLRA 100', '[BOLD] SLRA 200', '[BOLD] SLRA 200', '[BOLD] SLRA 300', '[BOLD] SLRA 300', '[BOLD] HPCA 100', '[BOLD] HPCA 100', '[BOLD] HPCA 200', '[BOLD] HPCA 200', '[BOLD] HPCA 300', '[BOLD] HPCA 300']
Experiments with symmetric window of five and ten context words are run to embed word representations in a d-dimensional vector, with d={100,200,300}. Except for some isolated results, performance is always much better with the stochastic low-rank approximation approach than with a Hellinger PCA approach. Calculating the reconstruction error of both approaches confirms that the PCA fails somehow to properly reduce the dimensionality. For a reduction from 10,000 to 100 dimensions, the PCA reconstruction error is 532.2 compared with 440.3 for the stochastic low-rank approximation. is thus not an approach offering a complete guarantee of success. An approach to robustifying PCA must be considered. This is what we propose with the stochastic low-rank approximation which, moreover, ensures a low memory consumption. For a given dimension, a window of ten context words outperforms, in general, a window of five context words. This confirms once again the benefit of using a larger window of context. Performance are globally better with 300 dimensions, but performance with 200 dimensions is just slightly lower, or even better in certain cases. Finally, using a distribution of rare words instead of frequent words (i.e. scenario (4) instead of scenario (1) here) has only an impact on the semantic word analogy task.