Evaluating semantic models with word-sentence relatedness
Table 3: Correlations with judgment-based rankings across all Target Concepts. The WordSpot and RelSpon baseline models are shown in gray; correlations for these baseline models reflect the 95% Confidence Interval of performance based on a 1000 iteration Monte Carlo simulation.
['[BOLD] System', '[BOLD] Correlation with Behavior ( [ITALIC] ρ)']
All models except for the n-gram model unambiguously outperformed the word-spotting baseline, but only the Ebiquity and Word2Vec systems outperformed the relationship-spotting benchmark system. These results suggest that all models except the n-gram model are capable of conceptual inference beyond simple word-spotting. The sub-baseline performance of the n-gram model might marginally improve with a larger corpus, but the interpretation would not change: higher-order models which can encode more abstract contextual information are consistently more predictive of human judgments as compared to the lower-order models that encode only surface information about local context. Indeed, both the Word2Vec and Ebiquity models consistently ranked sentences containing words related to the target above sentences that did not contain any related words. Given their beyond-benchmark performance, these models also appear capable of inferring some additional and useful conceptual structure.