Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain
Table 4: ASOIF Word Intrusion dataset: Accuracy of various word embedding models on selected word intrusion task sections, and total accuracy.
['Task Section', 'family-siblings', 'names-of-houses', 'Stark clan', 'free cities', 'Total']
The dataset includes 11180 task units in 13 sections. Due to the lower task complexity accuracy numbers are in the range of 70% to 83%, with best results for w2v-default with 83.01%. with a smaller window size, which suggests that for this task setup a smaller and more focused word context has benefits.