Lexical Semantic Recognition
Table 3: PARSEME test set results (%). (Gold): gold POS/lemmas (used in constraints only). (Pred.): predicted POS/lemmas. Nerima et al. (2017): a rule-based system which performed best for English in the shared task (Ramisch et al., 2018). Taslimipoor et al. (2019), Rohanian et al. (2019): more recent results on the test set (both used ELMo and dependency parses; only some scores were reported).
['[EMPTY]', '[BOLD] MWE-based  [BOLD] P', '[BOLD] MWE-based  [BOLD] R', '[BOLD] MWE-based  [BOLD] F', '[BOLD] Token-based  [BOLD] P', '[BOLD] Token-based  [BOLD] R', '[BOLD] Token-based  [BOLD] F']
While using GloVe embeddings does not reach the performance of the existing PARSEME systems—some of which were trained on the PARSEME training set—replacing them with BERT embeddings approaches the state of the art MWE-based F-score and exceeds the best reported token-based F-score, despite the challenging zero-shot out-of-domain generalization setting. This demonstrates that pre-training contextualized embeddings on large corpora can help models generalize to out-of-domain settings.