Parsing as Pretraining
Table 2: Same scores reported in Table 1 on the ptb and en-ewt ud, but using instead a (task-specific) 2-layer bilstm decoder (freezing and fine-tuning the pretraining network).
['[BOLD] Vectors', 'ptb lstm', 'ptb lstm-ft', 'en_ewt lstm', 'en_ewt lstm', 'en_ewt lstm-ft', 'en_ewt lstm-ft']
We first discuss the performance of the ff and ff-ft models. Among the precomputed word vectors, window-based approaches perform slightly better than alternatives such as GloVe, but they all do clearly surpass the control method (random embeddings). With respect to contextualized ones, for ff, both ELMo and BERT get a decent sense of phrase structures, with BERT slightly superior. This is due to the capacity of these decoders to obtain richer contextualized representations for the target task and to mask poor syntax capabilities of the input, even when the pretrained network is kept frozen.