Linguistic Knowledge and Transferability of Contextual Representations
Table 3: Performance (averaged across target tasks) of contextualizers pretrained on a variety of tasks.
['Pretraining Task', 'Layer Average Target Task Performance 0', 'Layer Average Target Task Performance 1', 'Layer Average Target Task Performance 2', 'Layer Average Target Task Performance Mix']
Bidirectional language modeling pretraining is the most effective on average. For example, when probing cwrs on the syntactic dependency arc classification (EWT) task, we see the largest gains from pretraining on the task itself, but with a different dataset (PTB). However, pretraining on syntactic dependency arc prediction (PTB), CCG supertagging, chunking, the ancestor prediction tasks, and semantic dependency arc classification all give better performance than bidirectional language model pretraining.