Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain
Table 2: ASOIF Analogies dataset (uni-grams): Accuracy of various word embedding models on selected analogy task groups, and total accuracy.
['Task Section', 'first-', 'child-', 'husband-', 'name-', 'houses-', 'Total']
In this section, we first discuss the results for the ASOIF dataset, and later compare those to the HP book series. Due to space limitations, we selected the results for five dataset sections, and the aggregated results. The total number of task units is 2848, the by far largest section are firstname-lastname analogy questions. The best results were provided by a word2vec model (w2v-ww12-i15-hs), but on this task, also GloVe is competitive. Both the w2v-default setting with its CBOW model and a small word window of 5, and the word2vec CBOW model with a larger window seem unsuited for this task configuration. FastText models yield lower accuracy than word2vec overall. Generally the accuracy is low with around 33.6%; the reasons for the difficulty of the task were already discussed above. lastname task, the results are high for such a simple baseline. This shows, that the characters last names tend to be very close to their first names in vector space.