Evaluation of Unsupervised Compositional Representations
Table 2: Pearson ρ for STS Benchmark and SICK relatedness, and Accuracy%/F1 for MSR Paraphrase detection. Results are shaded according to their statistical significance using the Williams test [Graham and Baldwin2014] with α=0.05. † pre-trained vectors used in the model above.
['[EMPTY]', '[EMPTY]', 'STS Benchmark  [ITALIC] ρ cosine', 'STS Benchmark  [ITALIC] ρ linear reg.', 'SICK  [ITALIC] ρ cosine', 'SICK  [ITALIC] ρ linear reg.', 'MSRP accuracy/F1 cosine', 'MSRP accuracy/F1 logistic reg.']
We highlight the best performance in each block; differences within the same shade are not statistically significant. Among the word embedding models, the subword skipgram si-skip achieved the best overall performance. For all models except si-skip, simple averaging performed poorly in semantic relatedness tasks, especially in the unsupervised setting, while sif-weighting generally outperformed idf-weighting (the improvement is most evident for GloVe). A similar trend is observed with random word vectors, which performed on par with doc2vec.