Segmentation-free Compositional n-gram Embedding
Table 1: Spearman rank correlations of the word similarity task on two different Chinese corpora. Best scores are boldface and 2nd best scores are underlined.
['[EMPTY]', 'skipgram [ITALIC] rich', 'sisg [ITALIC] rich', 'sembei', 'sembei-sum', '[BOLD] scne']
Results: As it can be seen, the proposed scne is competitive with or outperforms the baselines for both word and sentence similarity tasks. Moreover, it is worth noting that scne provides high-quality representations even when the size of training corpus is small, which is crucial for practical real-world settings where rich data is not available. As it can be seen, the performance of segmentation-dependent methods (skipgram, sisg) are decreased greatly by the noisiness of the corpus, while scne degrades only marginally. The other two segmentation-free methods (sembei, sembei-sum) performed poorly. This shows the efficacy of our method in the noisy texts. On the other hand, in preliminary experiments on English (not shown), scne did not get better results than our segmentation-dependent baselines and it will be a future work to incorporate easily obtainable word boundary information into scne for segmented languages.