What do you learn from context? Probing for sentence structure in contextualized word representations
Table 2: Comparison of representation models and their respective lexical baselines. Numbers reported are micro-averaged F1 score on respective test sets. Lex. denotes the lexical baseline (§ 4) for each model, and bold denotes the best performance on each task. Lines in italics are subsets of the targets from a parent task; these are omitted in the macro average. SRL numbers consider core and non-core roles, but ignore references and continuations. Winograd (DPR) results are the average of five runs each using a random sample (without replacement) of 80% of the training data. 95% confidence intervals (normal approximation) are approximately ±3 (±6 with BERT-large) for Winograd, ±1 for SPR1 and SPR2, and ±0.5 or smaller for all other tasks.
['[EMPTY]', '[BOLD] BERT-base  [BOLD] F1 Score', '[BOLD] BERT-base  [BOLD] F1 Score', '[BOLD] BERT-base  [BOLD] F1 Score', '[BOLD] BERT-base  [BOLD] Abs. Δ', '[BOLD] BERT-large  [BOLD] F1 Score', '[BOLD] BERT-large  [BOLD] F1 Score', '[BOLD] BERT-large  [BOLD] F1 Score', '[BOLD] BERT-large  [BOLD] Abs. Δ', '[BOLD] BERT-large  [BOLD] Abs. Δ']
We observe that ELMo and GPT (with mix features) have comparable performance, with ELMo slightly better on most tasks but the Transformer scoring higher on relation classification and OntoNotes coreference. Both models outperform CoVe by a significant margin (6.3 F1 points on average), meaning that the information in their word representations makes it easier to recover details of sentence structure. Furthermore, on all tasks except for Winograd coreference, the lexical representations used by the ELMo and GPT models outperform GloVe vectors (by 5.4 and 2.4 points on average, respectively). This is particularly pronounced on constituent and semantic role labeling, where the model may be benefiting from better handling of morphology by character-level or subword representations. We observe small absolute improvements (+6.3 and +3.5 for ELMo Full vs. Lex.) on part-of-speech tagging and entity labeling, but note that this is likely due to the strength of word-level priors on these tasks. Relative reduction in error is much higher (+66% for Part-of-Speech and +44% for Entities), suggesting that ELMo does encode local type information. How much information is carried over long distances (several tokens or more) in the sentence? To estimate this, we extend our lexical baseline with a convolutional layer, which allows the probing classifier to use local context. We see a similar trend on the other syntactic tasks, with 80-90% of ELMo performance on dependencies, part-of-speech, and SRL core roles captured by CNN ±2. Conversely, on more semantic tasks, such as coreference, SRL non-core roles, and SPR, the gap between full ELMo and the CNN baselines is larger. This suggests that while ELMo does not encode these phenomena as efficiently, the improvements it does bring are largely due to long-range information. This improves performance slightly, and changes our conclusion that ELMo was overall superior to GPT; the two are approximately equal on average, with slight differences on some tasks. To reduce noise, we report the average over five runs for experiments on Winograd coreference (DPR).