Learning Concept Abstractness Using Weak Supervision
Table 2: Correlation of abstractness scores assigned by the classifiers to manual annotations.
['test set', 'Naive Bayes', 'NN-RAD', 'RNN']
Evidently, the best results are obtained by the RNN classifier, yielding up to 0.740 correlation with human annotations. Notably, the simple Naive Bayes, utilizing only textual data, yields results of reasonable quality; the broad implication of this outcome lies in the potential applicability of this approach to resource-scarce scenarios where high quality word embeddings are not available. We attribute this improvement to the fact that fastText embeddings better capture morphological word properties and cover more extensive vocabulary. How many sentences containing a test concept suffice for a reliable prediction? We address this question by limiting the number of (randomly chosen) sentences used for rating. While the correlation obtained by RNN with 500 sentences containing a test concept reached 0.740 The plot in Figure LABEL:fig:sensitivity presents the correlation of the RNN and NB classifiers to label as function of number of (randomly sampled) sentences used for evaluation. Each such experiment (e.g., using 1, 5, 10 sentences) was averaged over 50 runs; the average correlation to label, as well as standard deviation, are plotted on the chart. The constant correlation yield by the (text-independent) NN-RAD algorithm is illustrated by the vertical line.