Segmentation-free Compositional n-gram Embedding
Table 4: Spearman rank correlations of the word similarity task on two different Japanese corpora.
['[EMPTY]', 'skipgram [ITALIC] rich', 'sisg [ITALIC] rich', 'sembei', 'sembei-sum', '[BOLD] scne']
In this section, we show the results of Japanese word similarity experiments. It contains 4427 pairs of words with human similarity scores. We omit sentence similarity task since there is no public widely-used benchmark dataset for Japanese yet. Following the main paper, given a set of word pairs and their human annotated similarity scores, we calculated Spearman’s rank correlation between the cosine similarities of the embeddings and the human scores. We use 2-fold cross validation for hyperparameters tuning. To see the effect of the noisiness of training corpora, we use two Japanese corpora, 100MB of Wikipedia corpus and 100MB of noisy SNS corpus (Twitter), which are also used in the Japanese noun category prediction task in the main paper.