Contextual Embeddings: When Are They Worth It?
Table 3: The performance (Matthews correlation coefficients) of BERT, random, and GloVe embeddings across the four linguistic categories defined by the GLUE diagnostic task: lexical semantics (LS), predicate-argument structure (PAS), logic (L), and knowledge and common sense (KCS). We also include the overall diagnostic performance.
['[BOLD] Category', '[BOLD] BERT', '[BOLD] Random', '[BOLD] GloVe']
GLUE Diagnostic Results The GLUE diagnostic task facilitates a fine-grained analysis of a model’s strengths and weaknesses in terms of how well the model handles different linguistic phenomena. The task consists of 550 sentence pairs which are classified as entailment, contradiction, or neutral. The GLUE team curated the sentence pairs to represent over 20 linguistic phenomena, which are grouped in four top-level categories: lexical semantics (LS), predicate-argument structure (PAS), logic (L), and knowledge and common sense (KCS). We follow the standard procedure and use the model trained on the MNLI dataset (using the random, GloVe, or BERT embeddings) to evaluate performance on the diagnostic task.