Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity
Table 3: Linguistic analysis on the Diagnostic dataset. The scores are R3 coefficients between gold and predicted labels, scaled by 100, for sentences containing linguistic phenomena of interest. We report all the coarse-grained categories: Lexical Semantics (LS), Predicate-Argument Structure (PAS), Logic (Lo), and Knowledge and Common Sense (KCS). Moreover, we report fine-grained categories for Lexical Semantics: Lexical Entailment (LE), Morphological Negation (MN), Factivity (Fa), Redundancy (Re), Named Entities (NE), and Quantifiers (Qu).
['[EMPTY]', 'Model', 'All', 'Coarse-grained LS', 'Coarse-grained PAS', 'Coarse-grained Lo', 'Coarse-grained KCS', 'Fine-grained LE', 'Fine-grained MN', 'Fine-grained Fa', 'Fine-grained Re', 'Fine-grained NE', 'Fine-grained Qu']
Fine-grained Analysis. To better understand how lexical information corroborates the model predictions, we perform a fine-grained analysis on the Diagnostic dataset Wang et al. As expected, Lexical Semantics is the category of phenomena that benefits the most (+43.7% for 1M iterations, +29.7% for 2M), but with significant gains also in phenomena related to Logic (+29.1% for 1M and +29.1% for 2M) and Knowledge & Common Sense (+51.7% for 1M). Interestingly, these results seem to suggest that knowledge about semantic similarity and lexical relations also partially encompasses factual knowledge about the world.