Lexical Semantic Recognition
Table 4: DiMSUM test set results (%). The test set covers three domains: reviews, tweets, and TED talks. (G): gold POS/lemmas (used in constraints only). (P): predicted POS/lemmas. Kirilin et al. (2016): the best performing system from Schneider et al. (2016). Kirilin et al. (2016) and other shared task systems had access to gold POS/lemmas and Twitter training data in addition to all of STREUSLE for training.
['[EMPTY]', '[BOLD] MWEs  [BOLD] P', '[BOLD] MWEs  [BOLD] R', '[BOLD] MWEs  [BOLD] F', '[BOLD] Supersenses  [BOLD] P', '[BOLD] Supersenses  [BOLD] R', '[BOLD] Supersenses  [BOLD] F', '[BOLD] Combined  [BOLD] Accuracy', '[BOLD] Combined  [BOLD] P', '[BOLD] Combined  [BOLD] R', '[BOLD] Combined  [BOLD] F']
Again, this is a zero-shot out-of-domain evaluation setting. In this case, the BERT model did not outperform the best shared task system, likely owing to the comparative difficulty of the full lexical semantic recognition task versus the restricted DiMSUM setting.