Not just about size - A Study on the Role of Distributed Word Representations in the Analysis of Scientific Publications
Table 1: Results from the word analogy and similarity tasks
['Algorithm', 'Dimensions', 'Corpus', 'Analogy Sem.', 'Analogy Synt.', 'Analogy Total', 'Word Sim. Spearman’s rho']
pre-trained embeddings in the analogy task and Spearmans’s rank correlation coefficient in word similarity. As expected, given the rather small size of the SciGraph corpus (886M tokens) compared to the other sources (number of tokens between 3B and 42B) and the fact that SciGraph focuses on the scientific domain, the performance obtained was significantly lower. A quick look at those benchmarks clearly shows that the SciGraph corpus does not cover all the vocabulary and semantic and syntactic relations that are evaluated in such tasks. However, as we will see below, the fact that SciGraph embeddings do not perform well in this task does not mean that they are not suitable for other tasks focused on the domain from which they were learned. Since word analogies and similarity are not fit, with the existing benchmarks, to evaluate SciGraph embeddings, we propose an extrinsic evaluation method in the form of a classification task.