Construction of a Japanese Word Similarity Dataset
Table 5: Variance of each dataset.
['Dataset', 'Variance']
WordSim353 comprises high-frequency words and so the variance tends to be low. In contrast, RW includes low-frequency words, unknown words, and complex words composed of several morphemes; thus, the variance is large. VSD has many polysemous words, which increase the variance. Despite the fact that our dataset, similar to the VSD and RW datasets, contains low-frequency and ambiguous words, its variance is 3.00. The variance level is low compared with the other corpora. We considered that the examples of the similarity in the task request reduced the variance level. At the beginning, we thought the context would serve to remove the ambiguity and clarify the meaning of word; however after looking into the dataset, we determined that the construction procedure used several extraordinary annotators. It is crucial to filter insincere annotators and provide straightforward instructions to improve the quality of the similarity annotation like we did.